Display control device, display control method, computer program product, and communication system

ABSTRACT

A control system, method and computer program product cooperate to assist control for an autonomous robot. An interface receives recognition information from an autonomous robot, said recognition information including candidate target objects to interact with the autonomous robot. A display control unit causes a display image to be displayed on a display of candidate target objects, wherein at least two of the candidate target objects are displayed with an associated indication of a target object score.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of and claims benefit of priorityunder 35 U.S.C. § 120 from, U.S. application Ser. No. 15/722,102, filedOct. 2, 2017, which is a continuation of and claims benefit of priorityunder 35 U.S.C. § 120 from, U.S. application Ser. No. 14/954,066, filedNov. 30, 2015, now U.S. Pat. No. 9,815,199, issued Nov. 14, 2017, whichis a continuation of and claims benefit of priority under 35 U.S.C. §120 from, U.S. application Ser. No. 14/701,152, filed Apr. 30, 2015, nowU.S. Pat. No. 9,802,311, issued Oct. 31, 2017, which is a continuationof and claims benefit of priority under 35 U.S.C. § 120 from, U.S.application Ser. No. 14/233,536, filed Jan. 17, 2014, now U.S. Pat. No.9,067,315, issued Jun. 30, 2015, which is a U.S. National Stage Entryunder § 371 of International Application No. PCT/JP2012/004744, filedJul. 25, 2012, which is based upon and claims benefit of priority under35 U.S.C. § 119 from prior file Japanese Patent Application No.2011-168922, filed Aug. 2, 2011, the entire contents of which beingincorporated herein by reference.

TECHNICAL FIELD

The present disclosure relates to a display control device, a displaycontrol method, a computer program product, and a communication system,for example, a display control device, a display control method, aprogram, and a communication system that display information for easilypredicting a behavior of an autonomous robot.

BACKGROUND ART

For example, generally, an autonomous robot autonomously acts (operates)in accordance with surroundings or general instruction from a user.

For example, if a user orders to bring a PET bottled tea withoutspecifying a brand name, the autonomous robot autonomously performs anaction that searches and brings a PET bottled tea in accordance with theinstruction.

That is, for example, when the autonomous robot finds a plurality of PETbottled teas, the autonomous robot selects a user-preferred PET bottledtea at its own judgment and brings the tea to the user.

Further, there is a technology with which in accordance with designationof goods and the destination of the goods, the user allows the robot tomove the goods to the destination (see, for example. Patent Document 1).

CITATION LIST Patent Document

-   Patent Document 1: Japanese Patent Application Laid-Open No.    2004-268148

SUMMARY

As recognized by the present inventor, even though the autonomous robotacts in accordance with the instruction from the user, the robot may actagainst the intention of the user due to the surroundings ormisinterpretation of the instruction of the user.

Therefore, the user preferably predicts the behavior of the autonomousrobot to some extent.

The present disclosure has been made to consider the above problems andmay easily predict the behavior of the autonomous robot.

Solutions to Problems

According to an exemplary control system for an autonomous robot,comprising:

-   -   an interface that receives recognition information from the        autonomous robot, said recognition information including        candidate target objects to interact with the autonomous robot;        and    -   a display control unit that causes a display image to be        displayed on a display of candidate target objects, wherein    -   the candidate target objects is displayed with an associated        indication of a target object score.

According to one aspect of the system,

the display image includes an overhead space diagram of a room thatincludes a location of the autonomous robot and respective locations ofthe candidate target objects.

According to another aspect of the system, the system further includes

the autonomous robot, wherein the autonomous robot includes in therecognition information the candidate target objects based on distanceto the respective candidate target objects.

According to another aspect of the system, the system further includes

the autonomous robot, wherein the autonomous robot includes in therecognition information a score for respective of the candidate targetobjects.

According to another aspect of the system, the autonomous robot includesin the recognition information object images of the candidate targetimages arranged in order of score.

According to another aspect of the system, the system further includes

the autonomous robot, wherein the autonomous robot includes in therecognition information space information regarding a space diagram ofthe room, and object images of the candidate target objects.

According to another aspect of the system, the system further includes

a control unit that receives user input and generates a command to theautonomous robot that provides user feedback to the autonomous robotregarding user selection of one or more of the candidate target objects.

According to another aspect of the system, the system further includes

the autonomous robot, wherein the autonomous robot is configured toidentify non-target objects.

According to another aspect of the system, the system further includes

the autonomous robot, wherein the autonomous robot is configured toidentify a default selection of one or more of the candidate targetobjects.

According to another aspect of the system, the system further includes

at least one of a tablet computer and a smartphone that includes theinterface and display control unit.

According to another aspect of the system, the system further includes

a scoring mechanism that identifies respective scores for the candidatetarget objects.

According to another aspect of the system,

-   -   the interface is configured to receive as input a category of a        target object, and transmit an indication of the category of        target object to the autonomous robot, and        the autonomous robot configured to identify within the scene one        or more target objects in the category.

According to another aspect of the system, the system further includes

-   -   the autonomous robot, wherein the autonomous robot assigns        degree information for candidate target objects, the degree        information being an indication of a likelihood of correct        detection of respective target objects in the category.

According to another aspect of the system,

-   -   the interface is configured to receive voice or gesture input        commands.

According to another aspect of the system, the system further includes

-   -   a display that displays candidate target objects identified by        the autonomous robot and user feedback sent via the        communications interface to assist in controlling the autonomous        robot.

According to a control method embodiment of the, the method includes

-   -   receiving through an interface recognition information from the        autonomous robot, the recognition information including        candidate target objects to interact with the autonomous robot;        and    -   displaying a display image on a display of candidate target        objects, wherein    -   at least two of the candidate target objects are displayed with        an associated indication of a target object score.

According to one aspect of the control method,

the displaying includes displaying an overhead space diagram of a roomthat includes a location of the autonomous robot and respectivelocations of the candidate target objects.

According to another aspect of the method, the method further includes

receiving user input and generating a command to the autonomous robotthat provides user feedback to the autonomous robot regarding userselection of one or more of the candidate target objects.

According to another aspect of the method, the method further includes

receiving voice or gesture input commands.

According to a non-transitory computer readable storage mediumembodiment, the storage medium includes instructions stored therein thatwhen executed by a processing circuit execute a control method for anautonomous robot, the method including

-   -   receiving through an interface recognition information from the        autonomous robot, the recognition information including        candidate target objects to interact with the autonomous robot;        and    -   displaying a display image on a display of candidate target        objects, wherein    -   at least two of the candidate target objects are displayed with        an associated indication of a target object score.

Effects of the Disclosure

According to the present disclosure, it is possible to easily predictthe behavior of the autonomous robot.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a view illustrating a configuration example of a robot controlsystem according to the present disclosure.

FIG. 2 is a view illustrating an example of a display screen on which aplurality of items is displayed.

FIG. 3 is a view explaining an outline of processing carried out by arobot.

FIG. 4 is a view illustrating an example of a recognition status of anobject.

FIG. 5 is a block diagram illustrating a configuration example of aninstructing device.

FIG. 6 is a first view illustrating an example of a display screen onwhich a display based on information from a robot is displayed.

FIG. 7 is a second view illustrating an example of a display screen onwhich a display based on information from a robot is displayed.

FIG. 8 is a third view illustrating an example of a display screen onwhich a display based on information from a robot is displayed.

FIG. 9 is a fourth view illustrating an example of a display screen onwhich a display based on information from a robot is displayed.

FIG. 10 is a fifth view illustrating an example of a display screen onwhich a display based on information from a robot is displayed.

FIG. 11 is a sixth view illustrating an example of a display screen onwhich a display based on information from a robot is displayed.

FIG. 12 is a seventh view illustrating an example of a display screen onwhich a display based on information from a robot is displayed.

FIG. 13 is an eighth view illustrating an example of a display screen onwhich a display based on information from a robot is displayed.

FIG. 14 is a ninth view illustrating an example of a display screen onwhich a display based on information from a robot is displayed.

FIG. 15 is a first view illustrating an example where the change inscores of an object is denoted by arrows.

FIG. 16 is a second view illustrating an example where the change inscores of an object is denoted by arrows.

FIG. 17 is a tenth view illustrating an example of a display screen onwhich a display based on information from a robot is displayed.

FIG. 18 is a flowchart for explaining display processing carried out byan instructing device.

FIG. 19 is a block diagram illustrating a configuration example of arobot.

FIG. 20 is a state transition diagram illustrating an example oftransition of a recognition status of an object.

FIG. 21 is a view for explaining a creating method that creates robotrecognition information in accordance with a recognition status of anobject.

FIG. 22 is a flowchart for explaining score calculation processingcarried out by a robot.

FIG. 23 is an eleventh view illustrating an example of a display screenon which a display based on information from a robot is displayed.

FIG. 24 is a twelfth view illustrating an example of a display screen onwhich a display based on information from a robot is displayed.

FIG. 25 is a thirteenth view illustrating an example of a display screenon which a display based on information from a robot is displayed.

FIG. 26 is a fourteenth view illustrating an example of a display screenon which a display based on information from a robot is displayed.

FIG. 27 is a fifteenth view illustrating an example of a display screenon which a display based on information from a robot is displayed.

FIG. 28 is a sixteenth view illustrating an example of a display screenon which a display based on information from a robot is displayed.

FIG. 29 is a seventeenth view illustrating an example of a displayscreen on which a display based on information from a robot isdisplayed.

FIG. 30 is an eighteenth view illustrating an example of a displayscreen on which a display based on information from a robot isdisplayed.

FIG. 31 is a view for explaining a feedback operation by a user.

FIG. 32 is a view illustrating an example where a recognition status ofan object is transited in response to the feedback operation.

FIG. 33 is a nineteenth view illustrating an example of a display screenon which a display based on information from a robot is displayed.

FIG. 34 is a twentieth view illustrating an example of a display screenon which a display based on information from a robot is displayed.

FIG. 35 is a twenty-first view illustrating an example of a displayscreen on which a display based on information from a robot isdisplayed.

FIG. 36 is a twenty-second view illustrating an example of a displayscreen on which a display based on information from a robot isdisplayed.

FIG. 37 is a twenty-third view illustrating an example of a displayscreen on which a display based on information from a robot isdisplayed.

FIG. 38 is a twenty-fourth view illustrating an example of a displayscreen on which a display based on information from a robot isdisplayed.

FIG. 39 is a flowchart for explaining feedback processing carried out byan instructing device.

FIG. 40 is a flowchart for explaining score recalculation processingcarried out by a robot.

FIG. 41 is a twenty-fifth view illustrating an example of a displayscreen on which a display based on information from a robot isdisplayed.

FIG. 42 is a twenty-sixth view illustrating an example of a displayscreen on which a display based on information from a robot isdisplayed.

FIG. 43 is a view illustrating an example where non-detected recognitionstate is transited into another state.

FIG. 44 is a view illustrating an example where a searching range or anobject is specified using a captured image.

FIG. 45 is a view illustrating an example where a searching range isspecified using an indoor space diagram.

FIG. 46 is a view illustrating an example where an amount of data of acaptured image transmitted from a robot to an instructing device isreduced.

FIG. 47 is a flowchart for explaining area specification processingcarried out by an instructing device.

FIG. 48 is a flowchart for explaining search processing carried out by arobot.

FIG. 49 is a block diagram illustrating a configuration example of acomputer.

DETAILED DESCRIPTION

Hereinafter, embodiments of the present disclosure (hereinafter,referred to as the present embodiment) will be described in thefollowing order.

1. First embodiment (an example where information for predictingbehavior of a robot is displayed on a display screen of an instructingdevice)

2. Modified example of the first embodiment

3. Second embodiment (an example where a feedback from a user is givento the robot using an instructing device)

4. Modified example of the second embodiment

5. Third embodiment (an example where a searching range of the robot isspecified using an instructing device)

1. First Embodiment

[Configuration Example of Robot Control System]

FIG. 1 illustrates a configuration example of a robot control system 1according to a first embodiment.

The robot control system 1 includes an instructing device 21 having adisplay screen 21 a, and an autonomous robot 22. The instructing devicemay be a remote controller with a display screen integrated into theremote controller (e.g., a smartphone, tablet computer, or laptopcomputer) or a remote controller that operates wirelessly with a remotedisplay device. For example, the instructing device, may use atelevision as the display device, and the instructing device serves asthe tactile interface for inputting instructions.

The instructing device 21 generates instruction information for causingthe robot 22 to do desired behavior (for example, behavior of bringing apredetermined object to a user) in response to manipulation of the userand transmits the information to the robot 22 through wirelesscommunication or the like. Moreover, the robot 22 may interact with thepredetermined object, where interact is not just taking images of theobject, but also contact, move, collect, perform electronic exchange ofinformation with, optical character recognition of, illuminant,purchase, label, or operate (such as turn on/off) the object.

For example, the instructing device 21, as shown in FIG. 2, displays aplurality of items such as “tea”. “soda”. “snack”, . . . which may beselected by a user on the display screen 21 a.

With reference to the display screen 21 a as shown in FIG. 2, among theplurality of items, the user selects (specifies) an item of object to bebrought by the robot 22. Further, for example, a category of the objectsto be brought by the robot 22 is displayed on the display screen 21 a asthe items.

In response to this, the instructing device 21 creates instructioninformation for bringing, as the target, the object belonging to thecategory (for example, “tea”) as the item selected by the user to theuser and transmits the information to the robot 22 by the wirelesscommunication or the like. Further, the instruction information includescategory information that represents categories as the items selected bythe user.

The robot 22 autonomously performs an operation of searching, as thetarget, the object belonging to the category as the item selected by theuser based on the instruction information from the instructing device21. The instructing device may be a remote controller with a displayscreen integrated into the remote controller (e.g., a smartphone, tabletcomputer, or laptop computer) or a remote controller that operateswirelessly with a remote display device. For example, the instructingdevice, may use a television as the display device, and the instructingdevice serves as the tactile interface for inputting instructions.

Further, a method for instructing the robot 22 is not limited to theabove-mentioned method that uses the instructing device 21. For example,if the robot 22 may recognize a voice of the user by the voicerecognition, the user may indicate the target using a voice. Theinstructing device may be a remote controller with a display screenintegrated into the remote controller (e.g., a smartphone, tabletcomputer, or laptop computer) or a remote controller that operateswirelessly with a remote display device. For example, the instructingdevice, may use a television as the display device, and the instructingdevice serves as the tactile interface for inputting instructions.

In this case, the robot 22 recognizes the instruction from the user as avoice, and autonomously performs an operation of searching the targetindicated by the user based on the recognition result. Further, forexample, if the robot 22 may recognize a gesture or posture of the user,the user may indicate the target by the gesture or posture.

The robot 22 is a two-legged robot that autonomously performs variousactions based on the surroundings or general instruction from the user(for example, general instruction to bring “tea”).

The robot 22 includes, as sensors that sense stimulus from the outside,a microphone corresponding to “ears” that sense sounds, and a cameracorresponding to “eyes” that sense light, which are provided inpredetermined positions. The details of the robot 22 will be describedbelow with reference to FIG. 19.

Next. FIG. 3 illustrates an outline of processing carried out by therobot 22.

FIG. 3 illustrates a surrounding image 31 (or a scene) obtained bycapturing the surroundings of the robot 22, an object image 32 thatindicates an object which is recognized as a target by the robot 22, anobject image 33 that indicates an object which is not recognized as atarget by the robot 22, an object image 34 that indicates an objectwhich is recognized as a possible target by the robot 22, and a partialimage 35 that indicates a part of a captured image obtained by capturingof the robot 22. Additionally, the scene provided by the robot 22 (orpre-stored in the controller 21), may be a computer generated graphicthat includes objects within the scene, such as furniture. Thus, animage and a computer generated graphic are two examples of a scene.

Further, the robot 22 freely moves, for example, in a room 36 in whichthe user resides to search a target.

While searching the target, the robot 22 appropriately transmits thesurrounding image 31 or the partial image 35 obtained by capturing witha built-in camera to the instructing device 21. Further, the surroundingimage 31 or the partial image 35 is used when the user specifies thesearching range of the robot 22 using the instructing device 21. Thiswill be mainly described with reference to FIGS. 43 to 47.

Further, the robot 22 moves in the room 36 and calculates a score thatindicates a degree that the object is the target, that is, a degree thatthe object is a subject to be processed by the robot 22, based on thecaptured image obtained by capturing the object with the built-incamera. In addition, an identifier for identifying (a functionindicating) whether the object is the target is used to calculate thescore.

The robot 22 determines the recognition status of the object based onthe score of the object. Further, hereinafter, it is described that therobot 22 uses the image obtained using the built-in camera to calculatethe score or transmits the image to the instructing device 21.

However, the image used to calculate the score or the image transmittedto the instructing device 21 is not limited to the image obtained by thebuilt-in camera in the robot 22. For example, an image obtained by acamera provided on the ceiling of the room 36 may be used. In this case,the camera provided on the ceiling of the room 36 transmits the capturedimage to the robot 22 by the wireless communication. Additionally, thescene provided by the robot 22 (or pre-stored in the controller 21), maybe a computer generated graphic that includes objects within the scene,such as furniture. Thus, an image and a computer generated graphic aretwo examples of a scene.

Next, FIG. 4 illustrates an example of the recognition status.

FIG. 4A illustrates two kinds of the recognition statuses of the robot22, which are “this is a target” and “this is not a target”.

FIG. 4B illustrates three kinds of the recognition statuses of the robot22, which are “this is a target”, “not sure yet”, and “this is not atarget”.

FIG. 4C illustrates four kinds of the recognition statuses of the robot22, which are “this is a target”, “this may be a target”, “not sureyet”, and “this is not a target”.

Further, in the first embodiment, as shown in FIG. 4C, the robot 22recognizes any one of the four recognition statuses.

For example, if the score of the object is equal to or higher than athreshold th1, the robot 22 recognizes that the object “is a target”. Ifthe score of the object is lower than the threshold th1 and equal to orhigher than a threshold th2 (<th1), the robot 22 recognizes that theobject “may be a target”.

Further, if the score of the object is lower than the threshold th2 andequal to or higher than a threshold th3 (<th2), the robot 22 recognizesthe object as “not sure yet” whether the object is a target. If thescore of the object is lower than the threshold th3, the robot 22recognizes that the object “is not a target”.

Here, the thresholds th1 to th3 are previously determined and thenstored in the robot 22. Further, the robot 22 compares the score withthe threshold in order to determine the recognition information of theobject, but the method of determining the recognition information is notlimited thereto.

Referring back to FIG. 3, for example, if it is judged that the score ofthe object “ABC tea” is equal to or higher than the threshold th1, therobot 22 displays on the display screen 21 a of the instructing device21 that the robot 22 recognizes that the object “ABC tea” “is a target”.

That is, for example, as shown in FIG. 3, on the display screen 21 a ofthe instructing device 21, the object image 32 indicating the object“ABC tea” is displayed in a state enclosed by, for example, aheavy-lined rectangle. The heavy-lined rectangle indicates that therobot 22 recognizes that the object “ABC tea” is “a target”.

Further, for example, as shown in FIG. 3, if it is judged that a scoreof an object “dog-shaped robot” is lower than the threshold th3, therobot 22 displays on the display screen 21 a of the instructing device21 that the robot 22 recognizes that the object “dog-shaped robot” is“not a target”.

That is, for example, as shown in FIG. 3, on the display screen 21 a ofthe instructing device 21, the object image 33 indicating the object“dog-shaped robot” is displayed with, for example, hatched lines. Thehatched lines indicate that the robot 22 recognizes that the object“dog-shaped robot” is “not a target”.

Further, for example, as shown in FIG. 3, if it is judged that a scoreof an object “orange juice” is lower than the threshold th1 and equal toor higher than the threshold th2, the robot 22 displays on the displayscreen 21 a of the instructing device 21 that the robot 22 recognizesthat the object “orange juice” “may be a target”.

That is, for example, as shown in FIG. 3, on the display screen 21 a ofthe instructing device 21, the object image 34 indicating a PET bottleon which the text “orange juice” is written is displayed in a stateenclosed by, for example, a thin-lined rectangle. The thin-linedrectangle indicates that the robot 22 recognizes that the object “orangejuice” “may be a target”.

Further, for example, if it is judged that a score of a predeterminedobject is lower than the threshold th2 and equal to or higher than thethreshold th3, the robot 22 displays on the display screen 21 a of theinstructing device 21 that the robot 22 recognizes that it is “not sureyet” whether the predetermined object is a target.

That is, for example, on the display screen 21 a of the instructingdevice 21, the object image indicating the predetermined object isdisplayed as it is (not enclosed by a rectangle or without hatchedlines). When the object image is displayed as it is, this indicates thatthe robot 22 is “not sure yet” whether the predetermined object is atarget.

As described above, the information recognized by the robot 22 isdisplayed on the display screen 21 a of the instructing device 21 sothat the user may easily predict the behavior of the robot 22.

However, the display method is not limited thereto, but as long as theuser instinctively figures out the behavior of the robot 22, any displaymethod may be used.

Further, by referring to the display screen 21 a, if it is judged thatan object that is not desirable as a target is recognized as a target,the user may give a feedback manipulation indicating that the object isnot a target to the instructing device 21.

In this case, the robot 22 recalculates the scores of the object inresponse to the feedback manipulation of the user, and autonomouslybehaves based on the score obtained by the recalculation. Further, thefeedback manipulation of the user will be described below mainly withreference to FIGS. 31 to 42.

[Configuration Example of Instructing Device 21]

Next, FIG. 5 illustrates the configuration example of the instructingdevice 21.

The instructing device 21 includes a manipulating unit 41, a controlunit 42, a display unit 43 having a display screen 21 a, a communicatingunit 44, and a storage unit 45.

The manipulating unit 41 includes manipulating keys or manipulatingbuttons. For example, if the user gives an instruction to the robot 22,the manipulating unit is manipulated by the user. When the manipulatingunit 41 is manipulated by the user, the manipulating unit 41 supplies amanipulating signal corresponding to the manipulation of the user to thecontrol unit 42.

The control unit 42 creates a robot recognition image including anobject image based on the robot recognition information from thecommunicating unit 44 and supplies the image to the display unit 43 todisplay the image on the display screen 21 a. Further, the robotrecognition information includes, for example, scores of the objects asinformation required to display on the display screen 21 a. In addition,the display example of the display screen 21 a will be described belowwith reference to FIGS. 6 to 17.

Further, the control unit 42 creates instruction information, feedbackinformation, or specified range information based on the manipulatingsignal from the manipulating unit 41 and supplies the information to thecommunicating unit 44.

Here, the feedback information refers to information indicating contentsfed back by the user in the instructing device 21.

The user uses the manipulating unit 41, for example, to feedback thatthe object recognized “as a target” by the robot 22 is not a target.This will be described below with reference to FIGS. 31 to 42.

Further, the specified range information indicates, for example, theposition of an area specified by the area specification of the useramong the entire area on the surrounding image 31 or the partial image35 displayed on the display screen 21 a of the instructing device 21.

The user uses the manipulating unit 41, for example, to perform an areaspecification for specifying the searching range where the robot 22searches the target among the entire area on the surrounding image 31 orthe partial image 35. This will be described below with reference toFIGS. 43 to 47.

The control unit 42 supplies robot recognition information from thecommunicating unit 44 to the storage unit 45 to be stored therein.

The display unit 43 displays the images on the display screen 21 a inaccordance with the control from the control unit 42.

Further, although the instructing device 21 is configured to include thedisplay unit 43, the instructing device 21 may be configured so as notto include the display unit 43. In this case, the control unit 42 may becoupled to the display unit 43 provided outside with a cable.

The communicating unit 44 receives the robot recognition informationtransmitted from the robot 22 and supplies the information to thecontrol unit 42. Further, the communicating unit 44 transmits theinstruction information, the feedback information, or the specifiedrange information from the control unit 42 to the robot 22 usingwireless communication or the like.

The storage unit 45 stores, for example, a control program which isexecuted by the control unit 42 in advance. Further, for example, thestorage unit 45 stores the robot recognition information from thecontrol unit 42.

[Display Example of Display Screen 21 a]

Next, FIGS. 6 to 14 illustrate an example of the display screen 21 awhen the instructing device 21 displays the robot recognition imagebased on the robot recognition information from the robot 22.

If the robot 22 is instructed to bring an object “tea” based on theinstruction information from the instructing device 21, the robot 22goes to search the object “tea”.

When the robot 22 explores the room but does not detect the object, therobot recognition information from the robot 22 is not transmitted tothe instructing device 21.

Therefore, as shown in FIG. 6, the control unit 42 displays a robotrecognition image in which nothing is captured on the display screen 21a, for example.

When the robot 22 detects an object “bird figurine” from the capturedimage obtained by capturing the inside of the room 36, the robot 22extracts an object image 51 representing the object “bird figurine” fromthe captured image and adds the object image 51 to the robot recognitioninformation to transmit the information to the communicating unit 44.

In response to this, the communicating unit 44 supplies the robotrecognition information from the robot 22 to the control unit 42. Thecontrol unit 42 supplies the object image 51 included in the robotrecognition information from the communicating unit 44 to the displayunit 43 to display the object image 51 on the display screen 21 a.

As a result, the object image 51, as shown in FIG. 7, is displayed onthe display screen 21 a at the left side of the drawing. Therefore, theuser may easily predict that the robot 22 detects the bird figurine asan object.

The robot 22 approaches the object “bird figurine” in order todiscriminate whether the object “bird figurine” is a target. Then, therobot 22 extracts the object image 51 representing the “bird figurine”from a captured image obtained by approaching and capturing the object“bird figurine” and adds the object image 51 to the robot recognitioninformation to transmit the information to the communicating unit 44.

In response to this, the communicating unit 44 supplies the robotrecognition information from the robot 22 to the control unit 42. Thecontrol unit 42 supplies the object image 51 included in the robotrecognition information from the communicating unit 44 to the displayunit 43 to display the object image 51 on the display screen 21 a.

As a result, as shown in FIG. 8, the object image 51 which is largerthan that in FIG. 7 is displayed on the display screen 21 a at the leftside of the drawing. Further, since the score of the object “birdfigurine” has not been calculated yet, the robot 22 is “not sure yet”whether the object “bird figurine” is a target. Therefore, in FIG. 8,the object image 51 is displayed as it is (not enclosed by a rectangleor without hatched lines)

The robot 22 calculates a score of the object “bird figurine” from thecaptured image obtained by approaching and capturing the object “birdfigurine”. Further, when the robot 22 recognizes that the object “birdfigurine” is “not a target” based on the calculated score, the robot 22transmits, to the communicating unit 44, the robot recognitioninformation including (information indicating) a recognition statusindicating that “the object is not a target”.

In response to this, the communicating unit 44 supplies the robotrecognition information from the robot 22 to the control unit 42. Thecontrol unit 42 creates the object image 51 with hatched lines based onthe recognition status included in the robot recognition informationfrom the communicating unit 44 and supplies the object image 51 to thedisplay unit 43 to be displayed on the display screen 21 a.

As a result, as shown in FIG. 9, the object image 51 with the hatchedlines is displayed on the display screen 21 a.

Further, the hatched lines added to the object image 51 represent thatthe robot 22 recognizes that the object “bird figurine” on the objectimage 51 is “not a target”.

Therefore, the user may easily predict that the robot 22 recognizes thatthe “bird figurine” on the object image 51 is “not a target”.

When the robot 22 detects an object “eighteen tea” from the capturedimage obtained by capturing the room, the robot 22 extracts an objectimage 52 representing the object “eighteen tea” from the captured image.

The robot 22 transmits the robot recognition information including theextracted object image 52 to the communicating unit 44.

In response to this, the communicating unit 44 supplies the robotrecognition information from the robot 22 to the control unit 42. Thecontrol unit 42 supplies the object image 52 included in the robotrecognition information from the communicating unit 44 to the displayunit 43 to display the object image 52 on the display screen 21 a.

As a result, as shown in FIG. 10, the object image 52 of the object“eighteen tea” detected by the robot 22 is displayed on the displayscreen 21 a at the left side of the drawing. Therefore, the user mayeasily predict that the robot 22 detects the object “eighteen tea”.

Further, the robot 22 calculates a score of the object “eighteen tea”from the captured image obtained by approaching and capturing the object“eighteen tea”. When the robot 22 recognizes that the object “eighteentea” is “a target” based on the calculated score, the robot 22transmits, to the communicating unit 44, the robot recognitioninformation including a recognition status indicating that the object“eighteen tea” is “a target”.

In response to this, the communicating unit 44 supplies the robotrecognition information from the robot 22 to the control unit 42. Thecontrol unit 42 creates the object image 52 enclosed by a rectangle andindicated by a triangle and the object image 51 with hatched lines basedon the robot recognition information from the communicating unit 44 andsupplies the object images 52 and 51 to the display unit 43 to bedisplayed on the display screen 21 a.

As a result, as shown in FIG. 11, the object image 52 enclosed by aheavy-lined rectangle and indicated by a triangle (▴ shown in FIG. 11)is displayed on the display screen 21 a at the left side of the drawingand the object image 51 with hatched lines is displayed at the furtherright than the object image 52.

Further, the heavy-lined rectangle that encloses the object image 52indicates that the robot 22 recognizes the object “eighteen tea” of theobject image 52 as “a target”. In addition, the triangle indicating theobject image 52 indicates that the robot 22 will return with the object“eighteen tea” of the object image 52.

Further, in this case, the control unit 42 allows the object imagesindicating the objects to be displayed from the left to the rightdirection of the display screen 21 a in the order of higher scores ofthe objects.

In addition, the display position of the object image corresponds to thescore of the object. Therefore, for example, the displaying intervalbetween a first object image and a second object image indicates thedifference in score between the objects of the first object image andthe second object image.

Accordingly, the user may instinctively notice the score of the objectindicated by the object image in accordance with the position of theobject image displayed on the display screen 21 a.

From the display screen 21 a shown in FIG. 11, the user may easilypredict that the robot 22 brings the object “eighteen tea”.

Further, when the robot 22 autonomously moves and detects a new objectfrom the captured image obtained by capturing using the built-in camera,as described above, the robot 22 creates robot recognition informationand transmits the information to the communicating unit 44.

In response to this, the control unit 42 supplies an object image to thedisplay unit 43 to display the object image on the display screen 21 abased on the robot recognition information from the communicating unit44.

As described above, the control unit 42 updates the display of thedisplay screen 21 a based on the robot recognition informationtransmitted from the robot 22 through the communicating unit 44.

That is, thereafter, for example, the display screen 21 a, as shown inFIG. 12, is updated to newly display an object image 53 representing anobject “ABC tea” in response to the behavior of the robot 22. Further,the display screen 21 a, as shown in FIG. 13, is updated to newlydisplay an object image 54 representing an object “C soda juice” and anobject image 55 representing an object “USB memory” in response to thebehavior of the robot 22. In addition, for example, the display screen21 a, as shown in FIG. 14, is updated to the one in which the positionsof the object images are changed in accordance with the scores inresponse to the behavior of the robot 22.

Further, on the display screen 21 a of FIG. 14, the object image 51 thatis displayed on the display screen 21 a of FIG. 13 is not displayed.

In FIG. 14, as an object is displayed on the farther left side of thedisplay screen 21 a, the score of the object is higher. Therefore, thescore of the object may be represented by the position at which theobject image is displayed.

Therefore, when the object images 52 to 55 are displayed in positionsaccording to the scores of the corresponding objects, the object image51 may not be displayed due to the size of the display screen 21 a orthe object images.

Further, in FIG. 14, the score of the object “bird figurine” of theobject image 51 is set to be significantly lower than the score of theobject “USB memory” of the object image 55.

In addition, in FIG. 14, the score of the object “ABC tea” of the objectimage 53 is set to be lower than the score of the object “eighteen tea”of the object image 52.

However, the triangle indicates the object “ABC tea” of the object image53 and the robot 22 intends to return with the object “ABC tea”. Thismeans that the robot 22 determines an object to bring in considerationof the position of the object or a user's preference in addition to thescore of the object.

Note that, in the first embodiment, as shown in FIGS. 6 to 14, theobject images are displayed in the order of higher scores of the objectsfrom the left to the right of the display screen 21 a side by side.However, the object images may be displayed in an arbitrary directionother than the rightward direction (for example, right to leftdirection).

Further, for example, in the first embodiment, as shown in FIGS. 6 to14, the object images are displayed in the order of higher scores of theobjects in an arbitrary direction side by side, and the object imagesare also displayed at positions corresponding to the scores of theobjects.

However, the object images may be displayed on the display screen 21 aonly using the order of the scores. Alternatively, for example, theobject images may be displayed on the display screen 21 a at positionscorresponding to the scores of the objects without using the order ofthe scores.

Further, for example, the object images may be displayed by changing atleast one of the order, the position, the size, the brightness, thedefinition, and the color of the object images in accordance with thescores of the objects.

In addition, for example, the scores of the objects may be displayed inassociation with the object images. Also, the scores, for example, maybe represented by a bar graph or a circle graph. When the score isrepresented by the circle graph, a percentage for a score of an objecthaving the highest score may be represented in the circle graph.

Further, for example, the changes in the scores of the objects may berepresented by an arrow in the display screen 21 a.

Specifically, for example, as shown in FIG. 15, if the score of theobject “C soda juice” corresponding to the object image 54 is changed tobe increased, a white arrow indicating the increase in the score of theobject “C soda juice” is associated with the object image 54 anddisplayed on the display screen 21 a.

Further, as shown in FIG. 15, if the score of the object “eighteen tea”corresponding to the object image 52 is changed to be decreased, a blackarrow indicating the decrease in the score of the object “eighteen tea”is associated with the object image 52 and displayed on the displayscreen 21 a.

In addition, for example, as shown in FIG. 15, if the score of theobject “USB memory” corresponding to the object image 55 is changed tobe decreased, a black arrow indicating the decrease in the score of theobject “USB memory” is associated with the object image 55 and displayedon the display screen 21 a.

When the contents as shown in FIG. 15 are displayed on the displayscreen 21 a, the user may predict that the robot 22 will not bring theobject “eighteen tea”, but bring the object “C soda juice” by mistake.In this case, before the robot 22 brings the object “C soda juice” bymistake, the user may clearly instruct to bring, for example, the object“eighteen tea”.

Further, when the contents as shown in FIG. 16 are displayed on thedisplay screen 21 a, the user may predict that the robot 22 will bringthe object “eighteen tea” which is desired by the user. In this case,the user does not instruct the robot 22 and may focus on his/her jobuntil the robot 22 brings the object “eighteen tea”. In addition, sinceFIG. 16 has the same configuration as FIG. 15, the description of FIG.16 will be omitted.

In addition, for example, on the display screen 21 a, the change in thescores of the objects may be represented by the bar graph or a linechart instead of an arrow. Alternatively, for example, together with thechange in the scores of the objects, or instead of the change in thescores of the object, a history of the scores may be displayed.

When the change or the history of the scores is displayed, the user mayfigure out the change or the history of the scores of the objects byreferring to the display screen 21 a and thus may easily predict thechange in the ranking of the scores and eventually predict the behaviorof the robot 22 in advance.

The above-mentioned display is realized by the control unit 42 referringto the robot recognition information stored in the storage unit 45.

Further, if the ranking of the scores is changed in the instructingdevice 21, the user may be notified of the change by sound, vibration,or blinking of a lamp.

In the first embodiment as shown in FIGS. 6 to 14, an object image of anobject that is recognized “as a target” by the robot 22 is enclosed by aheavy-lined rectangle, and an object image of an object that isrecognized “as a possible target” by the robot 22 is displayed as it is.

Further, for example, an object image of an object that is recognized as“not sure yet” by the robot 22 is enclosed by a thin-lined rectangle,and an object image of an object that is recognized as “not a target” bythe robot 22 is displayed with hatched lines.

However, the display method that represents the recognition statuses isnot limited to the rectangle or the hatched lines.

Further, for example, both the display method that represents therecognition statuses and a display method that represents the scores ofthe objects may be used. Alternatively, only one of the display methodthat represents the recognition statuses and the display method thatrepresents the scores of the objects may be used to display the objectimages on the display screen 21 a.

Next, FIG. 17 illustrates an example where detailed information of aselected object image is displayed as the user selects the object imageon the display screen 21 a.

On the display screen 21 a shown in FIG. 17, an object image 52 aindicating a bottom of the object “eighteen tea” (PET bottle), andobject images 52 b to 52 e obtained by capturing the object “eighteentea” from different directions as the detailed information of the object“eighteen tea” are displayed.

For example, the robot 22 extracts an object image from the latestcaptured image obtained by capturing the object as an object imagedisplayed on the display screen 21 a and adds the object image to therobot recognition information to transmit the information to theinstructing device 21.

Further, the robot 22 may use an object image extracted from a capturedimage having the highest resolution from a plurality of captured imagesin which the object is captured in addition to an object image extractedfrom latest captured image as an object image included in the robotrecognition information.

In addition, as the object image included in the robot recognitioninformation, among the plurality of object images, the robot 22 may usean extracted image that sufficiently represents the entire object or usea typical extracted image. Furthermore, as the typical extracted image,for example, an extracted image in which the object is captured at anangle of 45 degrees from above is considered.

For example, when the robot 22 obtains the captured image includingparts of the object images 52 b to 52 e by the previous capturing, andobtains a captured image including a part of the object image 52 a bythe latest capturing, the robot recognition image including the objectimage 52 a in which a bottom of the object “eighteen tea” is captured istransmitted to the instructing device 21.

Further, in the instructing device 21, as shown in FIG. 17, the objectimage 52 a in which the bottom of the object “eighteen tea” is capturedis displayed on the display screen 21 a. In this case, the user may notfigure out that the object represented by the object image 52 a is the“eighteen tea” by watching the object image 52 a displayed on thedisplay screen 21 a.

Therefore, the robot 22 transmits the robot recognition informationincluding the object images 52 b to 52 e in addition to the object image52 a to the communicating unit 44 of the instructing device 21. In thiscase, the communicating unit 44 supplies the robot recognitioninformation from the robot 22 to the control unit 42.

The control unit 42 allows, as shown in FIG. 17, an object image of anobject “glue stick”, the object image 52 a of the object “eighteen tea”,and an object image of an object “wallet” to be displayed on the displayscreen 21 a of the display unit 43 from the left side of the drawingbased on the robot recognition information from the communicating unit44.

Further, the control unit 42 supplies the robot recognition informationfrom the communicating unit 44 to the storage unit 45 to be storedtherein.

In the instructing device 21, when the user selects the object image 52a on the display screen 21 a using the manipulating unit 41, themanipulating unit 41 supplies a manipulating signal corresponding to theselecting operation of the user to the control unit 42.

Further, the control unit 42 reads the object images 52 b to 52 eincluded in the robot recognition information stored in the storage unit45 in accordance with the manipulating signal from the manipulating unit41 and supplies the images to the display unit 43 to be displayed on thedisplay screen 21 a.

Accordingly, on the display screen 21 a, as shown in FIG. 17, the objectimages 52 b to 52 e are displayed on the display screen 21 a as thedetailed information of the object “eighteen tea”.

Therefore, even though the user cannot figure out the object on theobject image 52 a from the object image 52 a, the user may easily figureout that the object captured in the object image 52 a is the “eighteentea” by the object images 52 b to 52 e displayed in response to theselecting operation of the user.

Further, for example, when the selecting operation of the user isperformed in accordance with the manipulating signal from themanipulating unit 41, the control unit 42 may display the enlargedobject image 52 a as detailed information. In this case, for example,the control unit 42 reads the object image 52 a included in the robotrecognition information stored in the storage unit 45 and enlarges theobject image with a predetermined enlargement ratio and displays theobject on the display screen 21 a.

Alternatively, for example, the control unit 42 may display athree-dimensional position where the object “eighteen tea” on the objectimage 52 a is detected on the display screen 21 a as the detailedinformation of the object image 52 a. In this case, the robotrecognition information including the three-dimensional position of theobject “eighteen tea” on the object image 52 a is stored in the storageunit 45 together with the object image 52 a.

Note that, for example, if the instructing device 21 is a personalcomputer to which a mouse is connected as the manipulating unit 41, theuser performs a selecting operation (mouseover) by moving a cursor onthe object image 52 a on the display screen 21 a using the mouse toselect the object image 52 a.

Further, only a predetermined number of object images corresponding tothe screen size of the display screen 21 a among the plurality of objectimages are displayed on the display screen 21 a. Therefore, only anobject image having a sufficiently high score, object images having topn-th scores, or an object image which is recognized as either “a target”or “a possible target” are displayed on the display screen 21 a.

Therefore, there is an object image which is not displayed on thedisplay screen 21 a, so that such an object image is preferablydisplayed on the display screen 21 a in response to the manipulation ofthe user.

That is, for example, the user performs a scroll operation that rotatesa scroll button provided in the mouse to reduce each object imagedisplayed on the display screen 21 a so as to display more object imageson the display screen 21 a.

In this case, as the manipulating signal in response to the scrolloperation is supplied from the manipulating unit 41, the control unit 42reads the robot recognition information stored in the storage unit 45.Also, the control unit 42 creates each object image reduced in responseto the scroll operation based on the read robot recognition informationand supplies the object images to the display unit 43 to be displayed onthe display screen 21 a.

Further, for example, the user may enlarge each object image displayedon the display screen 21 a in response to the scroll operation anddisplays fewer object images on the display screen 21 a.

In addition, the display screen 21 a, for example, displays the objectimage of score x1 at the center thereof, an object image having a scorehigher than the score x1 at the left side, and an object image having ascore lower than the score x1 at the right side.

On the display screen 21 a, an object image having a score x2 (<x1)lower than the score x1 may be displayed at the center of the displayscreen 21 a, an object image having a score higher than the score x2 maybe displayed at the left side, and an object image having a score lowerthan the score x2 may be displayed at the right side.

That is, for example, when the user moves the cursor to a position wherethe object image having the score x2 is displayed and manipulates themanipulating unit 41 so that the position is located at the centerposition of the display screen 21 a, the robot recognition imageallowing the object image having the score x2 to be displayed at thecenter of the display screen 21 a is created in the control unit 42 andthen displayed on the display screen 21 a of the display unit 43.

Further, for example, when the user manipulates the manipulating unit 41so as to locate the position where the object image having the score x1is displayed at the center of the display screen 21 a, the display ofthe display screen 21 a becomes the original display in which the objectimage having the score x1 is displayed at the center thereof.

In addition, for example, when the user enlarges a predetermined areaamong the entire area of the display screen 21 a to be displayed usingthe manipulating unit 41, the predetermined area on the display screen21 a is enlarged to be displayed.

[Description of Operation when Instructing Device 21 Displays RobotRecognition Image]

Next, display processing performed by the instructing device 21 will bedescribed with reference to a flowchart of FIG. 18.

For example, the display processing starts when the robot recognitioninformation is transmitted from the robot 22.

In step S21, the communicating unit 44 receives the robot recognitioninformation from the robot 22 in accordance with the control of thecontrol unit 42 and supplies the information to the control unit 42.

In step S22, the control unit 42 creates a robot recognition image basedon the robot recognition information from the communicating unit 44 andsupplies the image to the display unit 43 to be displayed on the displayscreen 21 a.

Further, the control unit 42 supplies the robot recognition informationfrom the communicating unit 44 to the storage unit 45 to be storedtherein.

In step S23, the control unit 42 judges whether the selecting operationhas been performed by the user in accordance with whether themanipulating signal has been supplied from the manipulating unit 41 inresponse to the selecting operation by the user.

Further, in step S23, if it is judged that the selecting operation hasnot been performed by the user, the control unit 42 returns theprocessing to step S21 and then repeats the same processing.

In addition, in step S23, if it is judged that the selecting operationhas been performed by the user in accordance with whether themanipulating signal has been supplied from the manipulating unit 41 inresponse to the selecting operation by the user, the control unit 42causes the processing to proceed to step S24. In this case, for example,it is assumed that the selecting operation for displaying the detailedinformation of the captured image 52 a shown in FIG. 17 has beenperformed.

In this case, in step S24, the control unit 42 reads the robotrecognition information stored in the storage unit 45 and supplies theobject images 52 b to 52 e included in the read robot recognitioninformation to the display unit 43 to be displayed on the display screen21 a. Thereafter, the processing returns to step S21 and then the sameprocessing is performed. Further, the display processing, for example,ends when the robot recognition information is not transmitted from therobot 22.

As described above, according to the display processing, the displayscreen 21 a is displayed as shown in FIGS. 6 to 17 based on the robotrecognition information from robot 22 so that it is possible to easilypredict the behavior of the robot 22.

Further, according to the display processing, in step S23, for example,as the selecting operation that selects the object image 52 a of FIG. 17is performed, in step S24, an object captured in the object image 52 ais displayed as the object images 52 b to 52 e as seen from differentdirections.

Therefore, the user may more precisely figure out an object of theobject image displayed on the display screen 21 a.

[Configuration Example of Robot 22]

Next, FIG. 19 shows a configuration example of the robot 22.

The robot 22 includes a communicating unit 61, a camera 62, a distancesensor 63, a microphone 64, a speaker 65, a control unit 66, a drivingunit 67, and a storage unit 68.

The communicating unit 61 receives instruction information, feedbackinformation, and specified range information from the instructing device21 and supplies the information to the control unit 66.

The communicating unit 61 transmits the robot recognition informationfrom the control unit 66 to the instructing device 21.

The camera 62 corresponds to “eyes” that sense light and includes, forexample, a CCD (charge coupled device) image sensor. The camera 62captures surroundings of the robot 22 and supplies a captured imageobtained by the capturing to the control unit 66.

The distance sensor 63 is a sensor that measures the distance from therobot 22 to the object and measures the distance to the object andsupplies the measured distance to the control unit 66.

The microphone 64 corresponds to “ears” that sense sound and collectsvoice and supplies a voice signal obtained by collecting the voice tothe control unit 66.

The speaker 65 corresponds to a “mouth” of the robot 22 and outputspredetermined voice in accordance with the control from the control unit66.

The control unit 66 controls the communicating unit 61 to the speaker65, and the driving unit 67. That is, for example, the control unit 66figures out the surrounding circumstances based on the captured imagefrom the camera 62, the distance from the distance sensor 63, or thevoice signal from the microphone 64 and controls the driving unit 67 inaccordance with the figured out surrounding circumstances.

Further, the control unit 66 controls the driving unit 67 based on theinstruction information from the communicating unit 61 and autonomouslyperforms the behavior instructed by the user. That is, for example, thecontrol unit 66 controls the driving unit 67 to allow the robot 22 toautonomously search, as a target, an object belonging to a categoryindicated by category information included in the instructioninformation.

In addition, the control unit 66 detects an object on the capturedimages based on the captured images from the camera 62 and extracts anobject image indicating the detected object. Furthermore, the controlunit 66 calculates a score of the detected object based on the capturedimage from the camera 62.

That is, the control unit 66, for example, reads an identifier of anobject which is a target from the storage unit 68 based on theinstruction information from the communicating unit 61. Further, thecontrol unit 66 extracts a feature amount indicating a feature of thedetected object from the captured images from the camera 62.

Further, the control unit 66 calculates the score of the detected objectusing the read identifier based on the extracted feature amount. Inother words, for example, the control unit 66 may use the identificationresult (score) obtained from the identifier as it is as a score of thedetected object or calculate a final score in time series manner byintegratively judging the identification result obtained from theidentifier.

The control unit 66 determines the recognition status of the objectbased on the calculated score of the object. Further, the determinationof the recognition status will be described below with reference to FIG.20.

Further, the control unit 66 determines an object image to be displayedon the display screen 21 a based on the calculated score of the objector the determined recognition status of the object. The method ofdetermining the object image to be displayed on the display screen 21 awill be described below with reference to FIG. 21.

The control unit 66, for example, changes an object image which isdetermined to be displayed on the display screen 21 a in accordance withthe recognition status of the object (for example, encloses with arectangle or adds hatched lines) and creates robot recognitioninformation including an object image after the change, a score of theobject, and display target information and supplies the information tothe communicating unit 61.

Further, the control unit 66 controls the distance sensor 63 to measurethe distance to the detected object. As a result, the distance sensor 63supplies the distance to the object detected by the control unit 66 tothe control unit 66.

The control unit 66 detects a three-dimensional position of the camera62 (robot 22) based on the captured image from the camera 62 or theposture of the camera 62 (position and orientation). Further, thedetails of a method of detecting the three-dimensional position of therobot 22 are disclosed in Japanese Patent Application Laid-Open No.2008-304268.

Further, the control unit 66 may use a positioning technology that uses,for example, a GPS (global positioning system) or Wi-Fi to detect thethree-dimensional position of the robot 22.

The control unit 66 calculates the three-dimensional position of thesensed object based on the detected three-dimensional position and thedistance supplied from the distance sensor 63, associates the positionwith the captured image in which the object is captured, and suppliesthe position to the storage unit 68 to be stored therein.

Further, the control unit 66 reads the captured image (for example, thesurrounding image 31 or the partial image 35 of FIG. 3) stored in thestorage unit 68 and creates a robot recognition image including even theread captured image to be supplied to the communicating unit 61.

The control unit 66 detects the specified area indicating an areaspecified by the specifying operation of the user based on the specifiedrange information from the communicating unit 61 and the captured imagestored in the storage unit 68.

Further, the control unit 66 reads the three-dimensional positionassociated with the object on the detected specified area from thestorage unit 68 and calculates a search range corresponding to thespecified area based on the read three-dimensional position.

In addition, the control unit 66 controls the driving unit 67 to driveportions corresponding to hands or feet of the robot 22 to perform anoperation of searching a target (object belonging to the categoryindicated by the category information) within the calculated searchrange.

The driving unit 67 drives the portions corresponding to “hands” or“feet” of the robot 22 in accordance with the control from the controlunit 66 so that the robot 22 autonomously behaves.

The storage unit 68, for example, holds a control program that isexecuted by the control unit 66 in advance and further stores (holds)data that is instructed to be written from the control unit 66.

Further, the storage unit 68 stores an identifier for identifying theobject for each object of a plurality of items. The identifier is afunction that has the feature amount of the object as an input andoutputs the score of the object. In addition, the identifier isgenerated by being learned in advance and stored.

In addition, the storage unit 68 stores the captured image (the capturedimage associated with the three-dimensional position of the object onthe captured image, for example, the surrounding image 31 or the partialimage 35) from the control unit 66.

Next, FIG. 20 shows an example of a transition diagram of therecognition status of the object.

In FIG. 20, as the recognition status that is recognized by the controlunit 66, “not detected (sensed)”, “not sure yet”, “this may be atarget”, “this is a target”, and “this is not a target” are shown.

The control unit 66 recognizes the object as any one status of “not sureyet”, “this may be a target”, “this is a target”, and “this is not atarget” in accordance with the calculated score of the object.

That is, for example, if the score of the object is equal to or higherthan the threshold th1, the control unit 66 recognizes that the object“is a target”. If the score of the object is lower than the threshold thand equal to or higher than a threshold th2 (<th1), the control unit 66recognizes that the object “may be a target”.

Further, for example, if the score of the object is lower than thethreshold th2 and equal to or higher than a threshold th3 (<th2), thecontrol unit 66 recognizes the status of object as “not sure yet”whether the object is a target. If the score of the object is lower thanthe threshold th3, the control unit 66 recognizes that the object “isnot a target”.

In addition, except when the feedback operation is performed by theuser, the control unit 66 recognizes that the object “is a target”always through the status that recognizes that this “may be a target”.

Accordingly, on the display screen 21 a, it is possible to prevent anobject that is recognized as “not sure yet” whether the object is atarget (corresponding to an object image enclosed by a thin-linedrectangle) from being suddenly changed into an object that is recognized“as a target” (corresponding to an object image enclosed by aheavy-lined rectangle).

Therefore, the user may pay attention to only an object that isrecognized on the display screen 21 a that “this may be a target”(corresponding to an object image displayed as it is) whether there isan object that does not correspond to a target.

As described above, when the object is recognized “as a target”, thecontrol unit 66 always goes through the status that recognizes that this“may be a target”. This is because, if an object that is recognized onthe display screen 21 a as “not sure yet” whether the object is a targetis suddenly changed into an object that is recognized “as a target”,this is inconvenient for the user because the user should pay attentionto all objects on the display screen 21 a.

Next, FIG. 21 illustrates an example of a determining method by whichthe control unit 66 determines an object image to be displayed on thedisplay screen 21 a in accordance with the score of the object.

In FIG. 21, an object 71, an object 72, an object 73, . . . an object 74are shown from the left side of the drawing in the order of higherscores.

The control unit 66 recognizes the object 71 “as a target” based on thecalculated score of the object 71. Further, the control unit 66recognizes that the object 72 “may be a target” based on the score ofthe object 72.

In addition, the control unit 66 recognizes the status of the object as“not sure yet” whether the object 73 is a target based on the score ofthe object 73. Furthermore, the control unit 66 recognizes that theobject 74 “is not a target” based on the score of the object 74.

For example, the control unit 66, as shown in FIG. 21, may determine todisplay the object images corresponding to top N objects having a highscore among the plurality of detected objects on the display screen 21 aof the instructing device 21.

Further, for example, the control unit 66, as shown in FIG. 21, maydetermine to display at least the object images corresponding to objectsrecognized as “this is a target” or “this may be a target” among theplurality of detected objects on the display screen 21 a of theinstructing device 21.

[Description of Operation when Robot 22 Calculates Score]

Next, with reference to a flowchart of FIG. 22, the score calculationprocessing that is performed by the robot 22 will be described.

The score calculation processing, for example, starts when the robot 22explores the room based on the instruction information from theinstructing device 21.

In step S41, the camera 62 captures the surroundings of the robot 22 andsupplies the captured image obtained by the capturing to the controlunit 66.

In step S42, the control unit 66 tries to detect an object on thecaptured image based on the captured image from the camera 62.

Further, when the object is detected from the captured image, thecontrol unit 66 causes the processing to proceed to step S43 based onthe captured image from the camera 62. In addition, when an object isnot detected from the captured image, based on the captured image fromthe camera 62, the control unit 66 returns the processing to step S41and then performs the same processing.

The control unit 66 judges a target indicating a subject to be processedby the robot 22 based on the instruction information supplied throughthe communicating unit 61 from the instructing device 21 and reads anidentifier for the judged target from the storage unit 68.

In step S43, the control unit 66 calculates a feature amount of thedetected object from the captured image from the camera 62. Next, thecontrol unit 66 calculates a score of the object based on the calculatedfeature amount using the identifier read from the storage unit 68.

In step S44, the control unit 66 creates robot recognition informationincluding the calculated score or the captured image stored in thestorage unit 68 and supplies the information to the communicating unit61.

In step S45, the communicating unit 61 transmits the robot recognitioninformation from the control unit 61 to the instructing device 21. Theprocessing returns to step S41 and then the same processing isperformed.

Further, the score calculating processing, for example, ends when therobot 22 finds the target and brings the object to the user.

As described above, according to the score calculation processing, thescore of the detected object is calculated and the robot recognitioninformation including the calculated score is transmitted to theinstructing device 21.

Therefore, on the display screen 21 a of the instructing device 21, therobot recognition image may be displayed as shown in FIGS. 6 to 17 sothat the user may easily predict the behavior of the robot 22.

2. Modified Example of First Embodiment

[Another Display Example of Robot Recognition Image]

Referring to FIGS. 6 to 17, the robot recognition images in which theobject having a higher score is displayed at farther left side of thedisplay screen 21 a have been described, but are not limited thereto.

Next, FIG. 23 illustrates another display example of the robotrecognition images.

In FIG. 23, on a upper screen 81 that indicates a screen located at theupper side of the display screen 21 a, as a score of an objectrepresented by an object image becomes higher, a robot recognition imagethat is disposed at farther left of the display screen 21 a isdisplayed. In this case, objects 91, 92, 93, and 94 have high scores indescending order.

Further, a latest captured image obtained by capturing with the camera62 is displayed on a lower screen 82 that represents a screen located atthe upper side of the display screen 21 a. As shown in FIG. 23, theobject 91 (“apple juice”) and the object 92 (“C soda juice”) displayedon the upper screen 81 are captured in the captured image.

On the display image 21 a, the object 91 on the upper screen 81 and theobject 91 (“apple juice”) on the lower screen 82 are connected by adotted line so that the correspondence relationship of the objects 91displayed on the upper screen 81 and the lower screen 82 may beunderstood.

The same applies to the object 92. Further, in this case, a latestcaptured image obtained by capturing with the camera 62 is included inthe robot recognition information.

That is, in FIG. 23, the latest captured image obtained by capturingwith the camera 62 is displayed on the lower screen 82, unlike the casedescribed with reference to FIGS. 6 to 17.

In FIG. 23, on the upper screen 81, the object images of the objects 91and 92 are enclosed by a heavy-lined rectangle. Therefore, it ispredicted that the robot 22 recognizes the objects 91 and 92 as thetarget. Further, referring to the lower screen 82 in FIG. 23, it ispredicted that the robot 22 exists near the objects 91 and 92.

Therefore, as compared with the case of referring to only the upperscreen 81 of FIG. 23, by referring to the upper screen 81 and the lowerscreen 82 of FIG. 23, the user may easily predict that the robot 22brings one of the objects 91 and 92 as a target.

Further, in FIG. 23, on the lower screen 82, the latest captured imageobtained by capturing with the camera 62 is displayed. However, forexample, a captured image obtained by capturing with the camera 62 at apredetermined time may be displayed. In addition, a composite image(image such as a panoramic view) created by composing a plurality ofcaptured images obtained by capturing with the camera 62 may bedisplayed.

In addition, for example, on the display screen 21 a, the captured imageobtained by capturing with the camera 62 may be displayed so as tooverlap a heavy-lined rectangle (for example, the apple juice on thecaptured image is enclosed by a heavy-lined rectangle). In this case, inthe display screen 21 a, an image that is displayed on the upper screen81 shown in FIG. 23 is not displayed, but only the captured image isdisplayed.

Further, for example, as shown in FIG. 24, instead of the latestcaptured image obtained by capturing with the camera 62, a space diagramshowing a space in the room may be displayed on the lower screen 82 ofthe display screen 21 a. In this case, the robot recognition informationincludes space information indicating the space diagram of the room. Inaddition, the space information, for example, is stored in the storageunit 68 of the robot 22 in advance.

In addition, as shown in FIG. 24, in the space diagram displayed on thelower screen 82, a path where the robot 22 moves from now is indicatedby a dotted line 101. In addition, in FIG. 24, the object displayed onthe upper screen 81 and the position on the space diagram in which theobject exists are connected by a dotted line, so that the correspondencerelationship can be understood.

Therefore, as compared with the case of referring to only the upperscreen 81 of the display screen 21 a of FIG. 24, by referring to theupper screen 81 and the lower screen 82 of the display screen 21 a ofFIG. 24, the user may predict that the robot 22 will return with theobject 91 (“apple juice”) as the target.

Further, when the display screen 21 a shown in FIGS. 23 and 24 isdisplayed, it is preferable to preferentially display the object presentnear the robot 22 (object present in a predetermined range from theposition of the robot 22) as the object image.

In addition, for example, on the display screen 21 a, the captured imageis displayed on the lower screen 82 shown in FIG. 23 and the spacediagram is displayed on the upper screen 82 shown in FIG. 24. However,for example, both the captured image and the space diagram may bedisplayed on the display screen 21 a.

In this case, for example, on the display screen 21 a, the space diagramincluding the dotted line 101 indicating the path where the robot 22moves from now is displayed together with the display shown in FIG. 23.

Next, FIG. 25 illustrates another display example of the robotrecognition image.

In FIG. 25, the latest captured image obtained by capturing with thecamera 62 of the robot 22 is displayed on the display screen 21 a.Objects 111 to 114 detected by the robot 22 are mainly captured in thecaptured images.

Further, in the objects 111 and 113, the behaviors to be performed withrespect to the objects by the robot 22 are displayed by balloons 111 ato 113 a.

The behavior “discard” is displayed in the balloon 111 a, the behavior“bring” is displayed in the balloon 112 a, the behavior “bring” isdisplayed in the balloon 113 a, and the behavior “discard” is displayedin a balloon 114 a. Further, when the display screen 21 a is displayedas shown in FIG. 25, processing information indicating processing (forexample, “discard”) of the robot 22 having an object (for example, “birdfigurine”) as a target to be processed is included in the robotrecognition information from the robot 22.

Further, the sizes of the balloons 111 a to 113 a, for example,correspond to the scores of the objects with the balloons. In addition,the score may be indicated by the color of the balloons instead of thesize of the balloons.

In the first embodiment, the user uses the instructing device 21 toinstruct the robot 22 to bring a “tea” as a target. However, theinstruction is not limited thereto. The user may instruct, for example,to discard the target in a trash bin.

For example, the user may use the instructing device 21 to instruct to“seize”, “pick up” or “clean up” the target.

For example, FIG. 25 is a display example that is displayed on thedisplay screen 21 a of the instructing device 21 when the user uses theinstructing device 21 to instruct the robot 22 to bring “juice” anddiscard a “small object” in the trash bin.

That is, the robot recognition image displayed on the display screen 21a may be displayed by any display method as long as the user mayinstinctively predict the behavior of the robot 22.

Further, for example, the user may use the instructing device 21 toselect a person as a target other than the “tea” as a target. In thiscase, the user may use the instructing device 21 to “call” a person as atarget.

In this case, for example, the detected person may be displayed on thedisplay screen 21 a together with the name of the person. Further, thename of the person, for example, is stored in the storage unit 68 of therobot 22 in advance in association with an identifier for identifyingthe person.

Further, if as the name of the detected person, a name of a differentperson is displayed on the display screen 21 a, the user may use themanipulating unit 41 to change the name into the name of the detectedperson.

For example, the instructing device 21, as shown in FIG. 26, allows therobot recognition image including object images 121 to 124 to bedisplayed on the display screen 21 a based on the robot recognitioninformation from the robot 22.

On the display screen 21 a shown in FIG. 26, it is displayed that therobot 22 recognizes that the object “eighteen tea” corresponding to theobject image 121 “may be a target”.

It is also displayed that the robot 22 is “not sure yet” whether theobject “C soda juice” corresponding to the object image 122 and theobject “potted plant” corresponding to the object image 123 are atarget, respectively and the robot recognizes that the object “stuffedanimal” corresponding to the object image 124 is “not a target”.

Therefore, the user may predict the behavior of the robot 22 asdescribed below by referring to the display screen 21 a shown in FIG.26. That is, for example, the user may predict that the robot 22eventually recognizes, “as a target”, the object “eighteen tea” that hasbeen recognized by the robot as “a possible target” and will bring theobject “eighteen tea”.

In this case, it is considered that the user may wait until the robot 22brings the object “eighteen tea”.

Further, for example, on the display screen 21 a shown in FIG. 27, aheavy-lined rectangle indicating that the robot 22 recognizes both theobject “ABC tea” corresponding to the object image 141 and the object “Stea” corresponding to the object image 142 as targets is displayed.

In addition, for example, on the display screen 21 a shown in FIG. 27, atriangle indicating that the robot 22 is going to return with the object“ABC tea” is displayed.

Therefore, the user predicts that the robot 22 is wondering which object“ABC tea” or “S tea” to bring but is going to tentatively return withthe object “ABC tea”.

If the object “ABC tea” or the object “S tea” makes no difference to theuser, the user does not need to manipulate the instructing device 21 andjust waits until the robot 22 brings the object “ABC tea” or the object“S tea”.

Further, for example, on the display screen 21 a of FIG. 27, since thetriangle is not displayed and the object “ABC tea” or the object “S tea”makes no difference to the user, if the user wants to have the objectquickly, it is preferable that the user instruct the robot 22 to bringthe object “ABC tea”.

Therefore, for example, the instructing device 21, as shown in FIG. 28,may indicate the object image 162 corresponding to the object “eighteentea” that the user wants to have by the user's manipulation, among theobject images 161 to 165 displayed on the display screen 21 a.

Further, in FIG. 28, for example, when the triangle is not displayed, asthe object image 162 corresponding to the object “eighteen tea” that theuser wants to have is indicated by the user's manipulation, the objectimage 162 enclosed by the heavy-lined rectangle and indicated by thetriangle is displayed on the display screen 21 a.

In addition, for example, if the triangle indicating the object “ABCtea” corresponding to the object image 182 among the object images 181to 187 is displayed on the display screen 21 a shown in FIG. 29, it ispredicted that the robot 22 returns with the object “ABC tea”.

If the user wants to have the object “S tea”, not the object “ABC tea”,the user manipulates the manipulating unit 41 of the instructing device21 to instruct the robot 22 to bring the object “S tea” corresponding tothe object image 184 instead of the object “ABC tea”.

That is, for example, as shown in FIG. 30, the user may specify theobject image 184 on the display screen 21 a using a pointer (arrow) toinstruct the robot 22 to bring the object “S tea” corresponding to theobject image 184.

As a result, on the display screen 21 a, instead of the triangle (FIG.29) indicating the object “ABC tea” corresponding to the object image182, a triangle (FIG. 30) indicating the object “S tea” corresponding tothe object image 184 is displayed.

3. Second Embodiment

[Example when User Who Watches Display Screen 21 a Instructs Robot 22]

Next, FIGS. 31 to 38 illustrate an example where the user performs afeedback operation for instructing the robot 22 by manipulating themanipulating unit 41 of the instructing device 21.

FIG. 31 illustrates an example where the user performs the feedbackoperation using the manipulating unit 41 of the instructing device 21 byreferring to the display screen 21 a of the instructing device 21.

In FIG. 31, the object images 201 to 204 displayed on the display screen21 a are shown. Further, the object images 201 to 202 are enclosed by arectangle indicating that the robot 22 recognizes the object “as atarget”.

Further, the object image 203 is displayed as it is, and the recognitionstatus of the robot 22 for the object “C soda juice” is “not sure yet”.In addition, hatched lines indicating that the recognition status of therobot 22 is “the object is not a target” are added to the object image204.

The user may use the manipulating unit 41 of the instructing device 21,for example, to perform a feedback operation that changes therecognition status of the object “C soda juice” indicated by the objectimage 203 from “not sure yet” into “this is a target” as shown by anarrow 221.

Further, for example, the user may use the manipulating unit 41 of theinstructing device 21 to perform a feedback operation that changes therecognition status of the object “eighteen tea” indicated by the objectimage 202 from “this is a target” into “this is not a target” as shownby an arrow 222.

The control unit 42 creates corresponding feedback information inaccordance with the manipulating signal from the manipulating unit 41corresponding to the feedback operation and supplies the information tothe communicating unit 44. The communicating unit 44 supplies thefeedback information from the control unit 42 to the robot 22.

The robot 22 recalculates the score of the object based on the feedbackinformation from the communicating unit 44 and changes the recognitionstatus of the objects based on the score obtained from the result.

Next, FIG. 32 illustrates an example where the control unit 66 of therobot 22 changes the recognition status of the object in accordance withthe feedback information from the instructing device 21.

For example, in the instructing device 21, if the user uses themanipulating unit 41 to perform the feedback operation, the manipulatingunit 41 supplies the manipulating signal corresponding to the feedbackoperation to the control unit 42.

The control unit 42 creates the feedback information based on themanipulating signal from the manipulating unit 41 and supplies theinformation to the communicating unit 44. The communicating unit 44transmits the feedback information from the control unit 42 to the robot22.

In this case, as shown in FIG. 31, for example, it is considered thatthe user uses the manipulating unit 41 of the instructing device 21 toperform the feedback operation that changes the recognition status ofthe object “C soda juice” indicated by the object image 203 from “notsure yet” into “this is a target” as shown by an arrow 221.

In this case, in the robot 22, the control unit 66 recalculates thescore of the object “C soda juice” based on the feedback informationsupplied through the communicating unit 61 from the communicating unit44 of the instructing device 21.

As a result, the control unit 66 determines (changes) the recognitionstatus of the object “C soda juice” from “not sure yet” into “this is atarget” as shown by an arrow 2211 based on a new score obtained byrecalculation.

Further, when the feedback operation that changes the recognition statusof the object from “this may be a target” into “this is a target” isperformed as a feedback operation, the control unit 66 performs thefollowing processing. That is, for example, the control unit 66determines the recognition status of the object from “this may be atarget” into “this is a target” as shown by an arrow 2212 based on thefeedback information from the communicating unit 61.

In addition, when the feedback operation that changes the recognitionstatus of the object from “this is not a target” into “this is a target”is performed as a feedback operation, the control unit 66 performs thefollowing processing. That is, for example, the control unit 66determines the recognition status of the object from “this is not atarget” into “this is a target” as shown by an arrow 2213 based on thefeedback information from the communicating unit 61.

Further, for example, as shown in FIG. 31, it is considered that theuser uses the manipulating unit 41 of the instructing device 21 toperform the feedback operation that changes the recognition status ofthe object “eighteen tea” indicated by the object image 202 from “thisis a target” into “this is not a target” as shown by the arrow 222.

In this case, as shown in FIG. 32, the control unit 66 determines therecognition status of the object “eighteen tea” from “this is a target”into “this is not a target” as shown by the arrow 222 based on thefeedback information from the communicating unit 61.

[Example of Feedback Operation]

Next, referring to FIGS. 33 to 38, an example where a user performs thefeedback operation by referring to the display screen 21 a of theinstructing device 21 will be described.

In the instructing device 21, the control unit 42 allows the robotrecognition image including, for example, object images 241 to 244 to bedisplayed on the display screen 21 a of the display unit 43, as shown inFIG. 33, based on the robot recognition information supplied through thecommunicating unit 44 from the robot 22.

Further, when the robot 22 recognizes an object “sports drink” indicatedby the object image 242 “as a target”, the control unit 42 allows therobot recognition image to be displayed on the display screen 21 a basedon the robot recognition information transmitted through thecommunicating unit 44 from the robot 22.

As a result, on the display screen 21 a, as shown in FIG. 34, a robotrecognition image including the object images 241 to 244 is displayed.

On the display screen 21 a shown in FIG. 34, it is predicted that therobot 22 recognizes, as a target, the object “sports drink” (an objectindicated by the object image 242), which is different from the object“eighteen tea” (object indicated by the object image 245) that the userwants to have.

Further, when the robot 22 behaves to bring the object “sports drink” asa target, the control unit 42 allows the robot recognition image to bedisplayed on the display screen 21 a based on the robot recognitioninformation transmitted through the communicating unit 44 from the robot22.

As a result, the display screen 21 a, as shown in FIG. 35, displays theobject image 242 added with a triangle indicating that the robot 22brings the object “sports drink” as a target.

In this case, as shown in FIG. 35, the user uses the manipulating unit41 of the instructing device 21 to perform the feedback operationindicating that the object “sports drink” is not a target. In this case,the manipulating unit 41 supplies the manipulating signal correspondingto the feedback operation of the user to the control unit 42.

The control unit 42 creates the feedback information based on themanipulating signal from the manipulating unit 41 and supplies theinformation to the communicating unit 44. The communicating unit 44transmits the feedback information from the control unit 42 to the robot22. In this case, for example, the robot 22 (control unit 66)recalculates a score of the object “sports drink” based on the feedbackinformation from the communicating unit 44.

As a result, the score of the object “sports drink” is lower than, forexample, the score of the object “eighteen tea” and the robot 22 behavesto bring the object “eighteen tea” as a target. Further, the robot 22changes the recognition status of the object “sports drink” from “thisis a target” to “this is not a target” based on the recalculated score.

In this case, in the instructing device 21, the control unit 42 allowsthe robot recognition image to be displayed on the display screen 21 abased on the robot recognition information transmitted through thecommunicating unit 44 from the robot 22. Further, the robot recognitioninformation includes at least the score recalculated based on thefeedback information in accordance with the feedback operation of theuser.

As a result, the display screen 21 a, as shown in FIG. 36, displays theobject image 245 added with the triangle indicating that the robotbrings the object “eighteen tea” as a target.

Further, the user may perform, for example, a feedback operationindicating that the object “eighteen tea” is a target, as shown in FIG.37, in addition to the feedback operation indicating that the object“sports drink” is not a target.

In this case, for example, the robot 22 (control unit 66) recalculatesthe score of the object “eighteen tea” based on the feedback informationfrom the communicating unit 44. As a result, the score of the object“eighteen tea” is higher than, for example, the score of the object“sports drink” and the robot 22 behaves to bring the object “eighteentea” as a target.

In contrast, the control unit 42 of the instructing device 21 allows therobot recognition image to be displayed on the display screen 21 a basedon the robot recognition information transmitted through thecommunicating unit 44 from the robot 22.

As a result, the display screen 21 a, as shown in FIG. 38, displays theobject image 245 added with the triangle indicating that the robotbrings the object “eighteen tea” as a target.

Further, for example, if it is indicated that the object is not a targetby the feedback operation, it is considered that the user uses themanipulating unit 41 to designate the object (object image thereof)which is not a target and make the designated object as a target.

In this case, the control unit 42 creates feedback informationindicating that the object designated by the user is not a target.

Further, for example, if it is indicated that the object is not a targetby the feedback operation, it is considered that the user uses themanipulating unit 41 to designate the object (object image thereof)which is a target and make the object that is not designated an objectthat is not a target.

In this case, the control unit 42 creates feedback informationindicating that the object that is not designated by the user is not atarget.

Further, the object may be designated by enclosing, with a frame, theobject on the display screen 21 a that is wanted to be designated. Inaddition, when the manipulating unit 41 includes a mouse, the user movesa cursor or the like to the object on the display screen 21 a to bedesignated and clicks the left button of the mouse (single clicking ordouble clicking) to designate the object.

In addition, if the instructing device 21 is a personal computer, and acontrol key and a shift key are provided as the manipulating unit 41,the control key and the shift key are used together to click the leftbutton to designate a plurality of objects.

Further, for example, after designating the object, the user may selectan option “this is not a target” among the displayed options “this is atarget”, “this is not a target”, and the like by clicking a right buttonof a mouse and perform the feedback operation indicating that thedesignated object is not a target.

In addition, for example, the user may click the designated object againwith the left button of the mouse to cancel the designation of theobject. In addition, for example, the user may select an object byclicking the left button to perform a dumping operation that dumps theobject outside the frame of the display screen 21 a as the feedbackoperation indicating that the object is “not a target”.

For example, when the manipulating unit 41 includes an “Enter” key, apressing operation that presses the “Enter” key while the object isdesignated may be adopted as the feedback operation indicating that thedesignated object is a target.

Further, for example, when the manipulating unit 41 includes a “Delete”key, a pressing operation that presses the “Delete” key while the objectis designated may be adopted as the feedback operation indicating thatthe designated object is not a target.

In addition, for example, when the manipulating unit 41 includes a “Tab”key, whenever the user presses the “Tab” key, the focus displayed on thedisplay screen 21 a may be moved.

The same applies to a case where the feedback operation indicating thatthe designated object is a target is performed.

[Regarding Feedback Processing Performed by Instructing Device 21]

Next, referring to the flowchart of FIG. 39, the feedback processingperformed by the instructing device 21 will be described.

Further, the feedback processing, for example, starts when the power ofthe instructing device 21 is turned on. In addition, the object image isdisplayed on the display screen 21 a.

In step S61, the control unit 42 judges whether the feedback operationby the user has been performed based on whether the manipulating signalcorresponding to the feedback operation has been supplied from themanipulating unit 41. The control unit 42 repeats the processing of stepS61 until it is judged that the feedback operation by the user has beenperformed.

In step S61, if it is judged that the feedback operation by the user hasbeen performed based on whether the manipulating signal corresponding tothe feedback operation has been supplied, the control unit 42 causes theprocessing to proceed to step S62.

In step S62, the control unit 42 creates the feedback information basedon the feedback operation by the user and supplies the information tothe communicating unit 44.

In step S63, the communicating unit 44 transmits the feedbackinformation from the control unit 42 to the robot 22 using the wirelesscommunication or the like and returns the processing to step S61 andthen repeats the same processing. Note that the feedback processing, forexample, ends when the power of the instructing device 21 is turned off.

As described above, according to the feedback processing, in accordancewith the feedback operation by the user, it is possible to indicate thatthe object displayed on the display screen 21 a is a target.

Therefore, for example, if the robot 22 recognizes an object that is nota target as a target, it is possible to feed back to the robot 22 thatthe object is not a target using the instructing device 21.

[Regarding Score Recalculation Processing that is Performed by Robot 22]

Next, referring to the flowchart of FIG. 40, the score recalculationprocessing performed by the robot 22 will be described.

The score recalculation processing, for example, starts when the powerof the robot 22 is turned on

In step S81, the communicating unit 61 repeats the processing of stepS81 until the feedback information from the instructing device 21 isreceived. In step S81, when the communicating unit 61 receives thefeedback information from the instructing device 21, the communicatingunit 61 supplies the received feedback information to the control unit66 and causes the processing to proceed to step S82.

In step S82, the control unit 66 recalculates the score of the objectbased on the feedback information from the communicating unit 61 andcauses the processing to proceed to step S83.

In step S83, the control unit 66 creates the robot recognitioninformation based on the recalculated score and supplies the informationto the communicating unit 61 and causes the processing to proceed tostep S84.

In step S84, the communicating unit 61 transmits the robot recognitioninformation from the control unit 66 to the instructing device 21 usingthe wireless communication or the like and returns the processing tostep S81 and then repeats the same processing. Further, the scorerecalculation processing, for example, ends when the power of the robot22 is turned off.

As described above, according to the score recalculation processing, thescore is recalculated based on the feedback information from theinstructing device 21 so that the score to which the feedback from theuser is reflected may be calculated.

4. Modified Example of Second Embodiment

In the second embodiment, referring to the display screen 21 a, the userperforms a feedback operation indicating that a predetermined object isa target or is not a target. However, the feedback operation by the useris not limited thereto.

That is, for example, when objects 261 to 263 are displayed, as shown inFIG. 41, on the display screen 21 a as one object, the robot 22recognizes the objects 261 to 263 as one object.

In this case, the user may use the manipulating unit 41 of theinstructing device 21, for example, to perform the feedback operationspecifying an area on which one object is displayed and allow the robot22 to recognize the objects 261 to 263 as one object.

Further, if the manipulating unit 41 includes a mouse, the feedbackoperation specifying an area on which one object is displayed, forexample, is performed using the mouse to specify the area on which oneobject is displayed and select an option “separate” to be displayed byclicking the right button.

Then, the control unit 42 creates the feedback information as themanipulating signal corresponding to the feedback operation is suppliedfrom the manipulating unit 41 and supplies the information to thecommunicating unit 44. The communicating unit 44 transmits the feedbackinformation from the control unit 42 to the robot 22 by the wirelesscommunication or the like.

In the robot 22, the communicating unit 61 supplies the feedbackinformation from the communicating unit 44 to the control unit 66. Thecontrol unit 66 recognizes each of the objects 261 to 263, which havebeen recognized as one object, as one object 261, 262, or 263 based onthe feedback information from the communicating unit 61.

Further, the control unit 66 calculates the scores of the objects 261,262, and 263 each recognized as one object and creates the robotrecognition information based on the calculation result and transmitsthe information to the instructing device 21 through the communicatingunit 61.

In the instructing device 21, the control unit 42 allows the robotrecognition image as shown in FIG. 42 to be displayed on the displayscreen 21 a of the display unit 43 based on the robot recognitioninformation transmitted through the communicating unit 44 from thecommunicating unit 61 of the robot 22.

Further, for example, when something that should be recognized as oneobject (for example, a set of knife, fork, and spoon that aresimultaneously used) is separated and displayed on the display screen 21a as a plurality of objects, the user may use the manipulating unit 41of the instructing device 21, for example, to perform the feedbackoperation specifying the area including the plurality of objectsseparated and displayed so as to allow the robot 22 to recognize theplurality of objects separated and displayed as one object.

In addition, when the manipulating unit 41 includes a mouse, thefeedback operation specifying the area including the plurality ofobjects separated and displayed, for example, designates the areas onwhich the objects are displayed using the mouse to select the option“merge” displayed by clicking the right button.

Note that, for example, in the second embodiment, the feedback operationis performed on the object that is displayed on the display screen 21 a.

However, for example, when the object on the captured image is notdetected by the control unit 66, by the area specifying operationspecifying an area including an undetected object on the captured image,the robot 22 may detect (recognize) the undetected object as an object.

As a result, the robot 22, as shown in FIG. 43, detects the unsensed(detected) object and determines the recognition status of the detectedobject from “undetected” to “not sure yet”.

That is, for example, the user may use the manipulating unit 41 of theinstructing device 21 to perform the area specifying operationspecifying the search range for the robot 22 to search the object toallow the robot 22 to search the object in the search range.

5. Third Embodiment

Next, FIG. 44 illustrates an example where the search range is specifiedusing the captured image.

For example, when the surrounding image 31 as shown in FIG. 44A isincluded in the robot recognition information as the captured image, thecontrol unit 42 of the instructing device 21 allows the surroundingimage 31 to be displayed on the display screen 21 a based on the robotrecognition information.

In this case, the user may use the manipulating unit 41 of theinstructing device 21, as shown in FIG. 44A, to perform the areaspecifying operation specifying a partial area 281 on the surroundingimage 31 displayed on the display screen 21 a as an area correspondingto the search range where the target is searched.

As a result, the manipulating unit 41 supplies the manipulating signalcorresponding to the area specifying operation of the user to thecontrol unit 42.

The control unit 42 creates the specified range information indicatingthe partial area 281 specified by the area specifying operation of theuser among all the areas on the surrounding image 31 in accordance withthe manipulating signal from the manipulating unit 41 and supplies theinformation to the communicating unit 44. The communicating unit 44transmits the specified range information from the control unit 42 tothe robot 22.

In contrast, the communicating unit 61 of the robot 22 receives thespecified range information from the communicating unit 44 of theinstructing device 21 and supplies the information to the control unit66. The control unit 66 detects the partial area 281 on the surroundingimage 31 stored in the storage unit 68 based on the specified rangeinformation from the communicating unit 61.

Then, the control unit 66 reads a plurality of three-dimensionalpositions associated with the objects on the partial area 281 from thestorage unit 68 and calculates (specifies) the search range forsearching the target based on the plurality of read three-dimensionalpositions.

Further, the calculating method by which the control unit 66 calculatesthe search range is not limited thereto. That is, for example, eventhough the three-dimensional positions are not associated with thepartial area 281, the control unit 66 may calculate the search rangefrom the partial area 281 or parameters indicating an effective viewingangle, an effective pixel number, a position, and an orientation of thecamera 62.

In this case, a first parameter indicating the position and theorientation of the camera 62 at a capturing timing is stored in thestorage unit 68 by the control unit 66 together with the capturingtiming when the surrounding image 31 is obtained by capturing at thetime when the camera 62 captures the image so as to be associated withthe surrounding image 31. Further, a second parameter indicating theeffective viewing angle and the effective pixel number of the camera 62is stored in the storage unit 68 in advance.

That is, for example, the control unit 66 reads the first parameterindicating the position and the orientation of the camera 62 at thecapturing timing when the surrounding image 31 is obtained by capturingand the second parameter indicating the effective viewing angle and theeffective pixel number of the camera 62 from the storage unit 68.

Then, the control unit 66 uses the first and second parameters tospecify an area on a real space captured as the partial area 281 in athree-dimensional viewing range corresponding to the partial area 281,that is, at the capturing timing, from the partial area 281 thatoccupies the surrounding image 31.

Further, as the search range, the area where the robot 22 can move isadopted among the viewing range.

The control unit 66 controls the driving unit 67 to drive the portionscorresponding to the hands or feet of the robot 22 and perform theoperation for searching the target in the calculated search range.

Further, for example, when the robot recognition information includesthe partial image 35 as shown in FIG. 44B, the control unit 42 of theinstructing device 21 displays the partial image 35 on the displayscreen 21 a based on the robot recognition information.

In this case, the user may manipulate the manipulating unit 41 of theinstructing device 21 to perform the area specifying operationspecifying the partial area 282 on the partial image 35 so as to detectthe object “apple juice” on the partial image 35 displayed on thedisplay screen 21 a.

As a result, the robot 22 performs the operation of searching the targetwithin the search range corresponding to the partial area 282 on thepartial image 35. Therefore, the robot 22 may detect the object “applejuice”.

Further, as the partial image 35, for example, a captured image obtainedwhen the area specified by the area specifying operation of the user iscaptured in a nearer range or a partial image of the captured imageobtained by capturing in the past may be adopted.

Next, FIG. 45 illustrates an example where the search range is specifiedusing the space diagram of the room.

For example, when the robot recognition information includes a spaceimage 301 as shown in FIG. 45 (image indicating a space diagram of theroom), the control unit 42 of the instructing device 21 displays thespace image 301 on the display screen 21 a based on the robotrecognition information. Further, if the storage unit 45 is configuredso as to store the space image 301 in advance, the control unit 42 readsthe space image 301 from the storage unit 45 and displays the image onthe display screen 21 a.

In this case, the user may manipulate the manipulating unit 41 of theinstructing device 21 to perform the area specifying operationspecifying a partial area 321 on the space image 301 displayed on thedisplay screen 21 a as the search range.

As a result, the manipulating unit 41 supplies the correspondingmanipulating signal to the control unit 42 in accordance with the areaspecifying operation of the user.

The control unit 42 creates the specified range information indicatingthe partial area 321 specified by the area specifying operation of theuser among all the areas on the space image 301 in accordance with themanipulating signal from the manipulating unit 41 and supplies theinformation to the communicating unit 44. The communicating unit 44transmits the specified range information from the control unit 42 tothe robot 22. Further, the specified range information may be used asspecific information to specify (figure out) the search range for therobot 22 to search the target.

In contrast, the communicating unit 61 of the robot 22 receives thespecified range information from the communicating unit 44 of theinstructing device 21 and supplies the information to the control unit66. The control unit 66 detects the partial area 321 on the space image301 stored in the storage unit 68 based on the specified rangeinformation from the communicating unit 61.

Then, the control unit 66 reads a plurality of three-dimensionalpositions that are associated with sub areas configuring the partialarea 321 from the storage unit 68 and calculates the search range basedon the plurality of read three-dimensional positions.

The control unit 66 controls the driving unit 67 to drive the portionscorresponding to hands or feet of the robot 22 to perform an operationfor searching a target within the calculated search range. Further, thecontrol unit 66 performs an operation for searching an object belongingto the category indicated by the category information included in theinstruction information as a target within the search range based on theinstruction information from the communicating unit 61. Further, forexample, when the control unit 66 does not receive the instructioninformation through the communicating unit 61 from the instructingdevice 21, the control unit 66 allows the robot to autonomouslydetermine the target to perform the operation of searching within thesearch range.

As a result, the robot 22 performs the operation of searching a targetwithin the search range corresponding to the partial area 321 on thespace image 301.

Further, if the space image 301 is displayed on the display screen 21 a,when the user uses the manipulating unit 41, for example, to select apartial area 322 among all the areas on the space image 301, a capturedimage 341 obtained by capturing in the partial area 322 may be displayedon the display screen 21 a.

That is, for example, in the instructing device 21, the control unit 42supplies the robot recognition information that is transmitted throughthe communicating unit 44 from the robot 22 to the storage unit 45 to bestored therein.

Accordingly, the captured image 341 is stored in the storage unit 45 soas to be associated with the partial area 322 on the space image 301together with the space image 301 as shown in FIG. 45.

Therefore, when the user selects the partial area 322, the control unit42 reads the captured image 341 corresponding to the partial area 322selected by the user from the storage unit 45 and displays the image onthe display screen 21 a as shown in FIG. 45.

Then, the user may use the manipulating unit 41 to perform the areaspecifying operation for specifying a partial area 361 on the capturedimage 341 with respect to the captured image 341 displayed on thedisplay screen 21 a in the same manner as shown in FIG. 44B.

As a result, the robot 22 performs an operation for searching the targetwithin the search range corresponding to the partial area 361 on thecaptured image 341.

Further, the captured image 341 is transmitted from the robot 22 to theinstructing device 21 in a state included in the robot recognitioninformation.

However, the amount of data for the robot recognition informationtransmitted from the robot 22 to the instructing device 21 is preferablysmall in order to avoid the convergence of data in the wirelesscommunication or the like.

Accordingly, for example, when the robot 22 transmits the captured image341 to be included in the robot recognition information, a low qualitycaptured image 341′ as shown in FIG. 46A which is obtained by reducingthe amount of the data of the high quality captured image 341 shown inFIG. 46B may be transmitted. Further, the captured image 341′ is notused for specifying the area by the user, but used, for example, whenthe object image of the object “apple juice” is displayed on the displayscreen 21 a. In addition, preferably, the captured image used forspecifying the area by the user is transmitted as a high quality imageas it is and displayed on the display screen 21 a.

Since among all the areas on the captured image 341′., an area 381 wherethe object (PET bottle of “apple juice”) is captured is used as theobject image, the captured image 341′ becomes a high quality image andother area than the captured image 341′ (a portion indicated by hatchedlines) becomes a low quality image.

That is, for example, the partial area 381 becomes a high resolutionarea or a color area and other area than the partial area 381 becomes alow resolution area or monochrome area.

[Regarding Area Specifying Processing Performed by Instructing Device21]

Next, referring to the flowchart of FIG. 47, the area specifyingprocessing performed by the instructing device 21 will be described.

In step S101, the communicating unit 44 receives the robot recognitioninformation from the robot 22 and supplies the information to thecontrol unit 42. Further, the robot recognition information, forexample, includes the surrounding image 31 as the captured image.

In step S102, the control unit 42 supplies the surrounding image 31included in the robot recognition information from the communicatingunit 44 to the display unit 43 to be displayed on the display screen 21a of the display unit 43.

In step S103, the control unit 42 judges whether the area specifyingoperation has been performed by the user in accordance with whether themanipulating signal corresponding to the area specifying operation ofthe user has been supplied from the manipulating unit 41. Further, thearea specifying operation, for example, refers to an operation thatspecifies a predetermined partial area 281 among all the areas on thesurrounding image 31 displayed on the display screen 21 a.

In step S103, the control unit 42 repeats the processing of step S103until it is judged that the area specifying operation of the user hasbeen performed in accordance with the manipulating signal from themanipulating unit 41.

In step S103, when it is judged that the area specifying operation ofthe user has been performed, the control unit 42 causes the processingto proceed to step S104. In this case, a manipulating signalcorresponding to the area specifying operation of the user is suppliedfrom the manipulating unit 41 to the control unit 42.

In step S104, the control unit 42 creates the specified rangeinformation indicating the partial area 281 on the surrounding image 31specified by the area specifying operation in accordance with themanipulating signal from the manipulating unit 41 and supplies theinformation to the communicating unit 44. Further, the specified rangeinformation is used as specific information to specify (figure out) thesearch range for the robot 22 to search the target.

In step S105, the communicating unit 44 supplies the specified rangeinformation from the control unit 42 to the robot 22 using the wirelesscommunication or the like and allows the robot 22 to search within thesearch range. By doing as described above, the area specifyingprocessing ends.

As described above, according to the area specifying processing, thesearch range in which the robot 22 searches the target may be specifiedby the area specifying operation by the user. Therefore, for example, itis possible to allow the robot 22 to search a target within a rangedesired by a user.

Specifically, for example, by specifying a range including at least anobject that is not detected by the robot 22 as a search range by thearea specifying operation by the user, it is possible to allow the robot22 to search the undetected object.

[Regarding Search Processing Performed by Robot 22]

Next, referring to the flowchart of FIG. 48, the search processingperformed by the robot 22 will be described.

In step S121, the communicating unit 61 receives the specified rangeinformation from the communicating unit 44 of the instructing device 21and supplies the information to the control unit 66. Step S121 may beperformed along with steps S122 and S123 which will be described below.That is, for example, the communicating unit 61 may receive thespecified range information from the communicating unit 44 of theinstructing device 21 while performing the processing of steps S122 orS122.

In step S122, the control unit 66 detects the partial area 281 on thesurrounding image 31 stored in the storage unit 68 based on thespecified range information from the communicating unit 61.

Then, the control unit 66 reads a plurality of three-dimensionalpositions associated with the sub areas configuring the partial area 281from the storage unit 68 and calculates (specifies) the search rangebased on the plurality of read three-dimensional positions.

In step S123, the control unit 66 drives the driving unit 67 so that therobot 22 can search the target in the calculated search range. Further,the control unit 66, for example, makes the object belonging to thecategory indicated by the category information included in theinstruction information as a target based on the instruction informationfrom the communicating unit 61. By doing as described above, the searchprocessing ends.

As described above, in the search processing, the target is searchedwithin the search range specified by the area specifying operation ofthe user.

Therefore, for example, when a range including at least an object thatis not detected by the robot 22 is specified as a search range by thearea specifying operation by the user, the robot 22 may detect theundetected object.

Further, the search range is specified by not the object itself, but thearea specifying operation of the user because it is difficult todesignate the object that is not detected by the robot 22.

Therefore, the user may allow the robot 22 to search the specifiedsearch range to detect the undetected object by specifying the searchrange including the object that is not detected by the robot 22.

Further, in step S103 of the area specifying processing, for example,the user specifies a predetermined partial area 281 among all the areason the surrounding image 31 displayed on the display screen 21 a tospecify the search range. However, the method of specifying the searchrange is not limited thereto.

That is, for example, when the instructing device 21 may recognize agesture or a posture of a user (hereinafter, referred to as gesture andthe like) from the captured image obtained by capturing the user, theuser may specify the search range by the gesture and the like.

Further, for example, if the instructing device 21 may recognize avoice, the user may specify the search range by the voice. In this case,for example, the user may specify the search range by speaking “kitchen”or “my room” as the search range.

In this case, for example, the user may designate (instruct) a categoryof the object to be a target by the voice, the gesture and the like. Asa result, in the instructing device 21, instruction informationincluding category information indicating a category designated by theuser is created.

In addition, for example, when the robot 22 may recognize the gestureand the like from the captured image obtained by capturing the user, theuser may directly specify the search range with respect to the robot 22by the gesture and the like.

In this case, for example, the control unit 66 recognizes the gesture ofthe user based on the captured image from the camera 62 and obtains(creates) the recognition result as specific information used forspecifying the search range. Then, the control unit 66 uses the obtainedspecific information to specify the search range and controls thedriving unit 67 to allow the robot 22 to search a target in thespecified search range.

Further, for example, when the robot 22 may recognize a voice, the usermay directly specify the search range with respect to the robot 22 bythe voice.

In this case, for example, the control unit 66 recognizes the voice ofthe user based on the voice of the user input from the microphone 64 andobtains (creates) the recognition result as the specific information.Then, the control unit 66 uses the obtained specific information tospecify the search range and controls the driving unit 67 to allow therobot 22 to search a target in the specified search range.

In this case, additionally, for example, the user may designate(instructs) the category of the object to be a target directly withrespect to the robot 22 by the voice, the gesture and the like.

As a result, in the control unit 66, instruction information includingcategory information indicating the category specified by the user iscreated and the robot 22 searches the object belonging to the categoryindicated by the category information included in the createdinstruction information as a target.

As the instructing device 21, for example, a personal computer may beused.

Further, the technology may have the following configuration.

-   (1) A control system for an autonomous robot, comprising:    -   an interface that receives recognition information from the        autonomous robot, said recognition information including        candidate target objects to interact with the autonomous robot;        and    -   a display control unit that causes a display image to be        displayed on a display of candidate target objects, wherein    -   the candidate target objects is displayed with an associated        indication of a target object score.-   (2) The control system of (1), wherein    the display image includes an overhead space diagram of a room that    includes a location of the autonomous robot and respective locations    of the candidate target objects.-   (3) The control system of (1), further comprising:    the autonomous robot, wherein the autonomous robot includes in the    recognition information the candidate target objects based on    distance to the respective candidate target objects.-   (4) The control system of (1), further comprising    the autonomous robot, wherein the autonomous robot includes in the    recognition information a score for respective of the candidate    target objects.-   (5) The control system of (4), wherein the autonomous robot includes    in the recognition information object images of the candidate target    images arranged in order of score.-   (6) The control system of (2), further comprising:    the autonomous robot, wherein the autonomous robot includes in the    recognition information space information regarding a space diagram    of the room, and object images of the candidate target objects.-   (7) The control system of (l), further comprising:    a control unit that receives user input and generates a command to    the autonomous robot that provides user feedback to the autonomous    robot regarding user selection of one or more of the candidate    target objects.-   (8) The control system of (7), further comprising:    the autonomous robot, wherein the autonomous robot is configured to    identify non-target objects.-   (9) The control system of (7), further comprising:    the autonomous robot, wherein the autonomous robot is configured to    identify a default selection of one or more of the candidate target    objects.-   (10) The control system of (1), further comprising:    at least one of a tablet computer and a smartphone that includes    said interface and display control unit.-   (11) The control system of (1), further comprising:    a scoring mechanism that identifies respective scores for said    candidate target objects.-   (12) The control system of (1), wherein    -   the interface is configured to receive as input a category of a        target object, and transmit an indication of said category of        target object to the autonomous robot, and        the autonomous robot configured to identify within the scene one        or more target objects in said category.-   (13) The control system of (1), further comprising:    -   the autonomous robot, wherein the autonomous robot assigns        degree information for candidate target objects, the degree        information being an indication of a likelihood of correct        detection of respective target objects in said category.-   (14) The control system of (1), wherein    -   the interface is configured to receive voice or gesture input        commands.-   (15) The control system of (11), further comprising:    -   a display that displays candidate target objects identified by        the autonomous robot and user feedback sent via said        communications interface to assist in controlling said        autonomous robot.-   (16) A control method for an autonomous robot, comprising:    -   receiving through an interface recognition information from the        autonomous robot, said recognition information including        candidate target objects to interact with the autonomous robot;        and    -   displaying a display image on a display of candidate target        objects, wherein    -   at least two of the candidate target objects are displayed with        an associated indication of a target object score.-   (17) The method of (16), wherein    the displaying includes displaying an overhead space diagram of a    room that includes a location of the autonomous robot and respective    locations of the candidate target objects.-   (18) The method of (16), further comprising:    receiving user input and generating a command to the autonomous    robot that provides user feedback to the autonomous robot regarding    user selection of one or more of the candidate target objects.-   (19) The method of (16), further comprising    receiving voice or gesture input commands.-   (20) A non-transitory computer readable storage medium having    instructions stored therein that when executed by a processing    circuit execute a control method for an autonomous robot,    comprising:    -   receiving through an interface recognition information from the        autonomous robot, said recognition information including        candidate target objects to interact with the autonomous robot;        and    -   displaying a display image on a display of candidate target        objects, wherein    -   at least two of the candidate target objects are displayed with        an associated indication of a target object score.

Note that a series of processing described above may be executed byhardware or software. If the series of processing is executed by thesoftware, the program configuring the software may be installed in acomputer embedded in the dedicated hardware or a general purposecomputer that may execute various functions by installing variousprograms from a program recording medium. With regard to a processorused for executing programs, the processor circuitry may be one or moreprocessors (including CPUs, ASICs, PAL's, etc) working together orseparately.

[Configuration Example of Computer]

FIG. 49 is a block diagram illustrating a configuration example ofhardware of a computer that executes the above-mentioned series ofprocessing by a program.

A CPU (central processing unit) 501 executes various processing inaccordance with programs stored in a ROM (read only memory) 502 or astorage unit 508. A program or data that is executed by the CPU 501 isappropriately stored in a RAM (random access memory) 503. The CPU 501,the ROM 502, and the RAM 503 are connected to each other by a bus 504.

An input/output interface 505 is connected to the CPU 501 through thebus 504. An input unit 506 including a keyboard, a mouse, and amicrophone and an output unit 507 including a display and a speaker areconnected to the input/output interface 505. The CPU 501 executesvarious processing in accordance with the commands input from the inputunit 506. Thereafter, the CPU 501 outputs the processing result to theoutput unit 507.

The storage unit 508 connected to the input/output interface 505includes, for example, a hard disk and stores a program or various datato be executed by the CPU 501. A communicating unit 509 communicateswith external devices through a network such as the Internet or a localarea network.

Further, the program may be obtained through the communicating unit 509and then stored in the storage unit 508.

When a removable medium 511 such as a magnetic disk, an optical disk, amagnetic optical disk or a semiconductor memory is installed, a drive510 that is connected to the input/output interface 505 drives theremovable medium and obtains the program or data recorded therein. Theobtained program or data are transmitted to and stored in the storageunit 508 if necessary.

The recording medium that records (stores) a program that is installedin the computer and is in an executable state by the computer, as shownin FIG. 49, may include the removable medium 511 which is a packagemedium including a magnetic disk (including a flexible disk), an opticaldisk (including a CD-ROM (compact disc-read only memory), and DVD(digital versatile disc)), a magnetic optical disk (including MD(mini-disc)) or a semiconductor memory, the ROM 502 that temporally orpermanently stores the program or a hard disk that configures thestorage unit 508. If necessary the program may be recorded in therecording medium through the communicating unit 509 which is aninterface such as a router or a modem using a wired or wirelesscommunicating medium such as a local area network, the Internet, ordigital satellite broadcasting.

Further, in the present specification, the steps that disclose theabove-mentioned series of processing may be performed sequentially inaccordance with the described order. However, the steps may not beperformed sequentially, but may be performed in parallel orindividually.

In addition, in the present specification, the system refers to a wholeapparatus including a plurality of devices.

Furthermore, the present disclosure is not limited to the first to thirdembodiments, but various modifications are available without departingfrom the scope of the present disclosure.

REFERENCE SIGNS LIST

-   1 Robot control system-   21 Instructing device-   21 a Display screen-   22 Robot-   41 Manipulating unit-   42 Control unit-   43 Display unit-   44 Communicating unit-   45 Storage unit-   61 Communicating unit-   62 Camera-   63 Distance sensor-   64 Microphone-   65 Speaker-   66 Control unit-   67 Driving unit-   68 Storage unit

The invention claimed is:
 1. A method for operating an instructingdevice comprising: controlling reception of current location informationof a movable apparatus, reception of an image data of a captured imagethat is captured with a camera at the movable apparatus, and receptionof object information regarding a candidate target object detected bythe movable apparatus in the image, the current location informationbeing generated based on GPS; controlling a display to display a mapindicating the planned travel route of the movable apparatus and thecurrent location along the planned travel route of the movable apparatusbased on the current location information; generating feedbackinformation to indicate the candidate target object is correctlyidentified as a target object in the image based on the objectinformation, the feedback information relating to detected user gestureinformation that is received to control an operation of the movableapparatus; controlling the display to display an indicator to indicatethe target object in the display based on the feedback information; andcontrolling, with circuitry, transmission of the feedback information tothe movable apparatus to control an operation of the movable apparatus.2. The method according to claim 1, wherein the feedback information isgenerated based on a command input indicating a selection of the targetobject.
 3. The method according to claim 1, wherein the movableapparatus is controlled to move towards the target object.
 4. The methodaccording to claim 1, wherein the movable apparatus is controlled tomove within a moving range based on at least the location information.5. The method according to claim 1, wherein the transmitting includescontrolling transmission of the feedback information including aparameter to control operation of the movable apparatus.
 6. The methodaccording to claim 1, wherein the indicator is displayed adjacent to thetarget object within a display area of the display.
 7. The methodaccording to claim 1, wherein the indicator being at least a portion ofa rectangle.
 8. The method according to claim 1, wherein the indicatoris superimposed on the target object of the captured image.
 9. Themethod according to claim 1, wherein the controlling includescontrolling the display to display the map with the indicator based onthe current location information and the feedback information.
 10. Themethod according to claim 1, wherein the controlling the display furtherincludes controlling the display to display the image and the mapsimultaneously.
 11. A non-transitory computer readable medium having acomputer program recorded thereon, the computer program configured toperform a method when executed on a computer, the method comprising:controlling reception of current location information of a movableapparatus, reception of an image data of a captured image that iscaptured with a camera at the movable apparatus, and reception of objectinformation regarding a candidate target object detected by the movableapparatus in the image, the current location information being generatedbased on GPS; controlling a display to display a map indicating theplanned travel route of the movable apparatus and the current locationalong the planned travel route of the movable apparatus based on thecurrent location information; generating feedback information toindicate the candidate target object is correctly identified as a targetobject in the image based on the object information, the feedbackinformation relating to detected user gesture information that isreceived to control an operation of the movable apparatus; controllingthe display to display an indicator to indicate the target object in thedisplay based on the feedback information; and controlling, withcircuitry, transmission of the feedback information to the movableapparatus to control an operation of the movable apparatus.
 12. Thenon-transitory computer readable medium according to claim 11, whereinthe feedback information is generated based on a command inputindicating a selection of the target object.
 13. The non-transitorycomputer readable medium according to claim 11, wherein the movableapparatus is controlled to move towards the target object.
 14. Thenon-transitory computer readable medium according to claim 11, whereinthe movable apparatus is controlled to move within a moving range basedon at least the location information.
 15. The non-transitory computerreadable medium according to claim 11, wherein the transmittingincludes: controlling transmission of the feedback information includinga parameter to control operation of the movable apparatus.
 16. Thenon-transitory computer readable medium according to claim 11, whereinthe indicator is displayed adjacent to the target object within adisplay area of the display.
 17. The non-transitory computer readablemedium according to claim 11, wherein the indicator being at least apart of outline of a rectangle.
 18. The non-transitory computer readablemedium according to claim 11, wherein the indicator is superimposed onthe target object of the captured image.
 19. The non-transitory computerreadable medium according to claim 1, wherein the controlling includescontrolling the display to display the map with the indicator based onthe current location information and the feedback information.
 20. Thenon-transitory computer readable medium according to claim 1, whereinthe controlling the display further includes: controlling the display todisplay the image and the map simultaneously.
 21. An instructing devicecomprising: circuitry configured to: receive current locationinformation of a movable apparatus, reception of an image data of acaptured image that is captured with a camera at the movable apparatus,and reception of object information regarding a candidate target objectdetected by the movable apparatus in the image, the current locationinformation being generated based on GPS; control a display to display amap indicating the planned travel route of the movable apparatus and thecurrent location along the planned travel route of the movable apparatusbased on the current location information; generate feedback informationto indicate the candidate target object is correctly identified as atarget object in the image based on the object information, the feedbackinformation relating to detected user gesture information that isreceived to control an operation of the movable apparatus; control thedisplay to display an indicator to indicate the target object in thedisplay based on the feedback information; and control transmission ofthe feedback information to the movable apparatus to control anoperation of the movable apparatus.
 22. The instructing device accordingto claim 21, wherein the feedback information is generated by thecircuitry based on a command input indicating a selection of the targetobject.
 23. The instructing device according to claim 21, wherein themovable apparatus is controlled to move towards the target object. 24.The instructing device according to claim 21, wherein the movableapparatus is controlled to move within a moving range based on at leastthe location information.
 25. The instructing device according to claim21, wherein the circuitry is further configured to: transmit thefeedback information including a parameter to control operation of themovable apparatus.
 26. The instructing device according to claim 21,wherein the indicator is displayed adjacent to the target object withina display area of the display.
 27. The instructing device according toclaim 21, wherein the indicator being at least a part of an outline of arectangle.
 28. The instructing device according to claim 21, wherein theindicator is superimposed on the target object of the captured image.29. The instructing device according to claim 21, wherein the circuitryis configured to control the display to display the map with theindicator based on the current location information and the feedbackinformation.
 30. The instructing device according to claim 21, whereinthe circuitry is further configured to: control the display to displaythe image and the map simultaneously.